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BY THE COMPTROLLER GENERAL 
OF THE UNITED STATES 



Follow Through: 
Lessons Learned From 
Its Evaluation And Need To 
mprove Its Administration 



Office of Education 

'Department of Health, Education, 
" and Welfare^ 



This report points out that although Follow 
Through, an experimental program designed 
to find more effective approaches to teaching 
young diif8(fn fropft-Jow- income families, 
achievedYdifjfereng^ in\outcome between 
Follow Ybrpugn and non-Follow-Through 
children, prOTlems in'its initial design and inv 
plementa^^ will limit reaching statisti.cplly 
reliablg-^nclusions as to which approaches 
are successful. 

The report recalls the events which led to a 
reorganization__of the national evaluation of 
Follow Through and suggests ways to prevent 
similar happenings. 
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To the President ot the Senate and*tne 
Speaker of 'the House of Representatives 

, In thp.s report we assess the Follow I^rough program 
and its national evaluat iohv and suggest ways to improve its 
administraft ionj. - The progra^i is administered by the Office 
of Education, Department of Health, Education, and Welfare. 
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fjollow Through is the largest educational re- 
Ibpment program Bver undertaken, we h^ve 

(1) the results of its efforts in iden- 
uT'approaches to teaching childr^ from 
ies and (2) its accompli shrrjents in provj.d- 
ij^e service^ and in involving parents in pro- 



We made cur review ■ pur suant to the Budget and Account-, 



t>3l U.S.C 53), and the Accounting and Audit- 
or Ml U.S.C. 67) . 



We are s.ekiding copies of this report to the Director, 
Office of ManaSgement and Budget, and to the Secretary of 
Health, Educatiion, and Welfare. 



. Comptroller General 
of the United States 
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DIGEST 

FplloW Through is an experimental program* 
designed to find more effective approaches 
to teaching young children' from low-income 
families. 

Colleges, universities , and private educa- 
tional research prgjanizatibns developed 
model approaches to install in classrooms. 
The Department of Health, Education, arid 
Welfare's (hew s) Office of Education con- 
tracted for a national evaluation to assess 
effects of .the approaches. (See pp. 1 and 4.) 

GAO found problems ih the initial design and 
Snplementation of Follow Through and a need 
to improve .program and project administra- 
tion. 
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recommends 'that the Secre- 
ct the Office of Education 



According 
tary of H 
to: . " 



— Insure that future experimental programs 
are not designed apart from evaluation to 
maximize the diegre-e to which experimental 
, results will be statistically reliable. 
(See p. 26.)' 

— Translate problems encetintered during, the 
i ' Follov Through experiment into criteria 
*for designing and/administering future 
^ experiments, including among other things, 
.P'"S=^^g£lteria to prevent collecting massivev 
^da^^--^ program results before goals ana ^ 
* objectives are properly defined. (See 
p. 26.) 



•Establish, to th>v^tent practicable, per- 
formance standards jfqr each service com- 
ponent and -require local educational agen- 
cies to evaluate their pi 
meeting these' standards^ 



\ 
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— Develop and disseminate comprehensive 
guidance to the agencies an a recordkeep- 
ing systemV in"cluding type of records 
needed to. document performance and pro- 
vide** feedback on program results. (See 
p. 39.) 

* — Develop anh disseminate specific guidance 
to "local educational agencies and consult- 
ants on the information that^ should be in- 
cluded in reports to the Office of Educa- 
tion and on their format and content. • 
(See p. 39.) 

— DeveTopf in the early implementation stage 
of future experiment s> regulations and 
guidelines concerning project administra- 
•tion and monitoring. (S.ee p. 39.) 

—Provide guidance to local educational 
i agencies on the format of eligibility rec- 
ords and require them to periodically re- 
^ port actual enrollment data. (See p. 49;) 

— Document reasons for exempting local edu- 
cational agencies from program eligibil- 
ity requirements. (See p. 49,)' 

HEW'agreed with these recommendations,^ How- 
ever, GAO believes the action taken by I^JEW 
will not fully im'plement two of its recom- 
mendations. (Se^ p. 40.) 

Problems and areas needing improvement ijn- 
clude: 

<> 

— Follpw Through planners designed a frame- 
work« for large-scale experimentation and 
evaluation without first defining srpecific 
goals and objectives of the experiment. 
(See p. 6 . ) 

— Even though initial results from the re- 
vised evaluation show that the approaches 
achieved some differences in outcome be- 
tween Follow Through' and nbn-Follow* 
• Through children, pr-obl^ms in the experi- 
ment's initial design end implementation^ 
will limit the Office of Education's abil-^ 
ity to reach statistically ^reliable con- ' 
'Iclus^iahs as to which approaches are suc- 
c^^ful. (See pp. 16 and' 19.) 



•Although the ultimate intention is to 
disseminate and encourage use of Follow* 
Through approaches that prove successful , 
the Office of Education has not yet devel- 
oped appropr iiite plans to accomplish this. 
(See pp. 22 and 24.)^ 

•Nonini^tructional services needed to' be in- 
creased and made m6re uniform. (See 
pp. 27 and 33.) ' 

The Office of Education has not (1) estab- 
lished performance standards for each 
service compon^ent, ('2) provided guidance 
on the type of records needed to document 
project performance, oi\ (3) required gran- 
tees to formally evaluate their pro jects . 
(See pp. 31 and 33.) \ 

The Office of Education's monitoring re- 
ports on Follow Through projectd" did not 
specify program weaknesses with their 
probable caus^es and corrective* actions 
needed. (See pp. 35 and 36.) 

The Office of^ Education did not have ade- 
quate control over project compli(|ince witi^^ 
eligibility requirements. (See pp. 41, 
47, and 48.) ^. 
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CHAPTER 1 



INTRODUCTION 



Follow Through is a program for children in kindergarten 
through the third?> grade design'ed primarily to build upon 
^ai'ns made by children from low-income families previously 
enrolled ip Head Start or similar preschool, programs.l/ Fol- 
low Through was authorized in 1967 under title II of the 
Economic Opportunity Act of 1964, as amended (42 U.S.C. 2809), 
as a service program. It- uses school, community, and family 
resources in meeting the educational, physical, psychological, 
and social needs of children. . The program's major emphasis, 
however, has been on assessing the effectiveness of dif- 
ferent approaches for educating young children from low-income 
families., It is th^ largest educational research and devel- 
opment program ever undertaken, according to pE. 

' This report includes information on {!) the design of 
Follow Tl\rough as an experiment and OE's efforts to evaluate 
the results of the program -and disseminate information on 
successful Follow Through approaches, (2) pro ject . administra- 
tion, including the delivery of noninstruct.ional services 
and project compliance with funding requirements, and (3) the 
eligibility characteristics of the children enrolled in the 
nine projects reviewed. 

' PROGRAM ADMINISTRATION 

S t ■ 

lO a June 1967 memorandum of understanding, revJ.sed in' 
May 1969, the Director, Office of Economic Opportunity, del-' 
egated responsibility for administering Follow Through- to 
the Secretary of Health, Education, and Welfare (HEW). The 
Secretary assigned responsibility for the program to OE, 
which issued regulations on funding and developed a draft 
program manual. In March 1974 OE published proposed ^regula- 
tions covering all. Follow Through operations. Final regula- 
tions were issued on April 21, 1975, '^nd became ef-fective 
on June 5, 1975. 

OE asked State educational agencies and State economic 
opportunity offices to '^nominate communities to participate 
in Follow Through.. OB/th^n invited several nominated com- 
munities to submit project applications for funds and. 



ERIC. 



l/According to the Office of Education (OE), a similar pre- 
school 'program is one that offers the full range of com- 
prehensive services that Head Star t- of fers ; that is, in- 
structional, medical, dental, psychological, nutritional, 
. and social services. . . ^ 
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selejcted communities — 



As) — to participate. 
St grade school, J/ 




on the basis of these apMications 
primarily local educational ag 

The LEAS/ began their projects — ^ ^-^ , 

(either/ kindergarten or first-grade) and added ^ nW grade^/^ 
each^^ear thereafter through third grade. A 

Xx/p'e awaTrds 1-year grants directly to L^iM and^^r equips,... 
tfiem to suppleftiejjit the grants with local/^hds j/r noncash 
^.ntributions. The lo^/al, Qt non-Fed^r^^, sha/e ranges from 
^ 4 to 25 percent of^e amot^nt of tMf<>lipw, Through grant. 
At the time pf .our^'^sitewor 1^, LEAs were ^Iso jreqUixed to 
'supplement ^heir grants with a portion of tnei funds, they 
recje^ive<3 under the federal prpgragi of. aid .to educatiionally 
deprJLved children — title I of^-feh^ Elementary and Secondary 
Edification Act of 19,65, as amended /2Q u/s.C. *241a). The . 
title I contribution had to be\at/leas;t 15 percent of the 
combined Follow Through and title/l f^nds, except that 
no LEA was required to contributed mO'fre tban 10 percent 
of its 'total title I allocation yto i'olldw Through. 

To receive Federal funds, /leAs are obligated to macft 
the requirements in the projecyc application, including the' 
grant terms and conditions; the memorandum of unde.r standing ; 
and the draft program manual ^ Each LEA has a coordinator/ 
or director responsible for Merall project management 'in 
acfcbrdance with the above^^r^uirements. ^ ^ 

The following table , /prepared" from OE statistipsr / . " 
summarizes FoUow^Th rough, operations from inception through 
school year 1973^4. - 



/ 



School year 



Federal, funds^ 
available 

/(millions) 



Projects 



Childrej>^enr oiled 

from low-income 
families (note, a) 



1967- 68 

1968- 69 

1969- 70 

1970- 71 

1971- 72 
.1972-73 

'3-74 



$^^3.75 
11/. 25 
32»00 
/70.30 
' 69^.00 
63.06 
50.62 



39 
103 
-161 
Ip 
178 
173 
170 



2,900 
15,500 
37,000 
*60,200 
78,170 
84,000 
78,000 



a/OE/does not Collect information on the number of children 
eiirolled fronK-non-low- income families. 
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FOLLOW T-Hl^^UGH AS A SERVICE PROGRAM 



/ 



^ Head Start/ also ^author ized by the Economic Opportunity 
Act/ is a comprehensive program that provides to presj^rcr^lT 
pover ty<::hildren educational, medical, cjental, psycfioiogica;! 
nutr It^ional , and social services. In addition, each. Head';^ 
St^v<€ project .promotes parent ^nd community involvement. 
^yij^dts and provides staff training and development, 'because 
rarly. Head SJfeart evaluations indicated that program benefits, 
were soon dissipated if pot reinforced in the regular school^ 
system, tfte Congress created Follo>/ Through to sustain arfd 
$3^pand/4£he gains of Head Start graduates. Therefore, the 
"pto^ijy^m^diS designed to provirde the same kinds of Services 
ivities as Head Start. 



\ 



The size of FolloK^ Through? never reached the proportions 
ifitended. An approapiation of $120 million was requested 
for the 'first year i>f the program, and/ OE envisioned increas- 
in^lx^, greater amojumts for later years./ To accomplish the 
.pr^jc;a]^^ oiDjecjblve significant fund i/n^ was considered 
nece^ssary beqa^se, during the fiscal year before cireation of 
^ / Follow Throuj^tvTTJead Start had servedf about 215,000 children"^ 
in its full-year program and 466, 000 /children in its ^;3uipiner 
program at an annual cost of about $316 million.' /[^^ 
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as $15 miV 



Because this.** 



The first appropriation for Follow/Through 
lion and was to cover 2 years of operation/. Be< 
amount would serve only, a fraction- of the /eligible presglfbol 
children, the program's emphasis. .Was changed during sqKool year 
1967-68 from a* service program ^o an expe^rimentar^pr'j^^ram. 



FOLLOW THROUeS AS AN EXPERIMENTAL PROGRAM 



The purpose of Follow Through as (|in- ^e5cper i^filental program, 
referred to by OE as "planned variatiwy^ ^is ^a-«^evei<>^<^n4,^,, 
validate different approaches to educ^^King young children frofn 
lovi(-incbme^ families. To impleroe^nt the change, OE initially/ 
identified 14 approaches for use by LEAs. Starting with s<?hool 
year 1968-69, OE required LEAs entering the program to select 
and ^ implement 1 of the 14 approaches. LEAs that had ent^ed 

but not required, t^ 



the program In 1967 wer6 encouraged 
SjBlect one of the approaches^/ 
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During,' school year ;^9;73.:::.7|4.^,.,^^^^ were beting , 

implemented in Follow Through b^^'pjects lETigoughput the^ Nation. 
They gehei<ally conformed- to orfe bf the fpljLowing grqiips: 
(1) highl^ structured^ projects ertphasizing* acaqlemic^Skills «^ 
in reading and ari^hro^tic, (2) projects emphasizing emotipnal- 
revelopment^an<3,..ejicaux:,^,ging exploration and discovery 




are^s> 3)^.projectg stre$f'sing cog 
and/ an^i^^ing quest^iplis, prbbl 





tiye thinking 
solving, and 



wrffelng /^^hd^C pro jects /focusirfflf ojilprepar ing 



parents to improved-he educat^j 
"cKlldren. - - • 




elopment of their 



The approaches were dev/loped primarily by colleges, uni- 
versities, and private educational research organizations. 
Th6se institutions, referr/"d to as sponsors, contracted- with 
•OE and LEAS to provide cu/riculum materials, teacher, training, 
and other assistance needed to install the .approaches in the 
classrooms. OE provide^ basic sponsor sup^9rt through grants. 
In June 1968., it cont/ap-tetT^ith a firm /to .assess the effects 
of the approaches af^varioiis location^ /. 

None of" the noninstructional services were eliminated 
when the program's emphasis chang^ from service to experimental. 
Thus, Follow. Through became anyefxpe/imental program in a service 
setting. 

PHASEOUT OF FOLLOW Tt}ROU( 



Until the spring of 1972, OE's plan was to widely dissemin- 
ate information -aijaut-su^iaessful Follow Through approaches. ^' 
jOne important long-range objective under, "consideration was to 
-1 "incorporate the Follow Through cbncept and, design into the 
^//^ operat>bn-of programs under title I of the' Elementary and 
Secondary Education Ac^. / 

\ ixCVisi^lSll , however, OE changed these plans' and de- ' 

\ cjrd^d to phase out the program./ To accomplish this, OE 
\ / "^d planned to drop one' grade, each year / starting- with 
y ' kindergarten at the beginning of school year 1974-75 and 
X ending with the third grade upon the completion of school 
• ■ year 1976-77. However, the ^appropriation for fiscal year 
1975 indluded funds for a kindergarten class to start 
' during/school year '1974-75 and, acc^jfding to OE, the Congress 
also/plans to appropriate an amount ' for • fiscal year 1976 
tjj^t will include fund^ for a kindergarten class for -school 
year 19751-76. Therefore, the phaseout strategy for the pro- 



gram has -iiot yet been resolved. 
SCOPE OF REVIEW 

Qur review was made at OE* headquarters in Washington, 
D.C., and at nine selected project sites, one each in Alabama, 
Arkandasy Colorado, Florida, Georgia, Mississippi, Montana, 
Texas^ ahd Utah. We reviewed, the legislative history of 
Follow Through, OE policies and procedures, project appli- 
cations, \an«i other related documents. We also reviewed the . 
initial jrfesults of a national evaluation Follow Through and 
interviewed. Federal, State, and local bfficifals responsible 
for adminlste'ring and operating Follow', Throbgh . In addition, 
we obsefved classroom activities and intjei: viewed selected 
teacher^ and parents of Follow Through enrolVees. 

r . ■ • 

; 4 Ii<2 



Our review at >the project sites covered school years 
•1970-71 anca 1971-72. Projects reviewed included: (1) proj- 
ects with at least 2 years* of operating exper-.ience* and • 
<2) projects representing a .variety 6f model approaches. 
We also considered a project's size and whether .it was urban 
or rural. We believe that the problems noted are shared by 
many projects. Our findings, and' conclusions should not, 
however, .be. interpreted as necessarily being 'typical of all 
'Follow Through" projects. / 






. • . • • CHAPTER 2 

• FOLLOW THROUGH: AN EXPERIMENTAL PROGRAM 

Ft>llow Through represents a unique attempt to better 
the lives of children from low-income families through 
imaginative, instructional approaches and. a comprehensive 
program of health and social services. and parent activities. 
School year 1973-74 represents the programme seventh complete 
year of operatibn and the sixth year of a national evaluation 
Af the program. Although the evaluation was substantially 
revised during the fourth year of the 'exper iment , problems 
in both initial design and implementation of the experiment 
willv in the final analysis, limit OE's ability to reach 
statistically reliable conclusions as to which sponsor' ap- 
proaches are successful for teaching young disadvantaged 
children. These problems, which we believe cannot practi- 
cab^ly be overcome-, include 

— lack of random assignment of LEAs. to sponsors and 

— lack" of comparable control groups. ^ 

These problems, plus the OE contractor's reservations 
concerning design and measurement problems, raise iguestions 
about the experiment's dependability to judge the model ap- 
proaches. However, OE can capitalize oa this experience 
as lessohsUearned fox future experiments. - ' ^ 

Nevertheless, ini'tial results from the revised ev'alua- 
tion 3how that, on the basis of eight criterion measures,' 
sponsors have achieved some differences in outcome between 
Follow Through and non-Follow-Through children. After com- 
pletion of the evaluatipn, OE wants to disseminate and en- 
courage the use of approaches foirnd to be successful. How- 
ever, OE has not yet developecj plans to accomplish this. • ^ 



LESSONS LEARNED FROM FOLLOW THROUGH 



During its early years the national evaluation of Follow^ 
Through was strongly criticized by parties both in and out- 
side the Federal Government because it lacked a well-defined 
plan. This criticism resulted in a substantial reorganiza- 
,tion of the evalua4:ion during school year 197-l-72i^ Although 
a new evaluation strategy with specified objectives emerged 
from the reorganization, its overall effectiveness will, be 
limited because of problems in both the initial design and 
implemeritation o'f the experiment. 
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, Follow Through was, howevet^ a unique experiment 
ijivolving the evaluation of a three-way partnership among 
the school^ the sponsor, and the^home. Because OE lacked ex- 
perience in organizing such a program, it may not have rec- 
ognized the consequences of early decisions affecting the 
.program's d.esign and implementation. The putpase of this 
section, therefore, is to recall the events of the national 
evaluation 'which led to^ its reforganization and to suggest 
ways to prevent similar happenings. 

History and development of 
the national evaluation 

In the summer of 1968 OE awarded a $900,»000 contract 
to Stanford Research Institute, Henlo Park, California, to ' 
make a national evaluation of^ Follow Through. According to. 
the contract, the Institute's m^in tasks were to describe 
the various model approaches, collect data, and shape and ' 
define an evaluation strategy. The primary purpose for the 
evaluation, as stated in the Institute's proTposal, was to 
provide guidance on program development. 

— 11re~Thstitute's proposal tor the second evaluation year 

(scttooi yeat 1969-70) was approved, on June 25vl969, for 
abovit''^2,150,000 and was basically an extensroa of thfe first 
^ proposal in that it called for additional data' collection 
and further development of the evaluation plan. This pro- 
posal, prepared before the data collected in school year ' 
1968-6a had been fully analyzed, identified to some extent 
the emerging evaluation strategy^ This, Institute ' s plan was 
to Treasure changes in pupil_growth and achievement and par eat 
attitudes a.t successive times', using various tests and oth^r 
devaluation instruments, and to compare these results with 
.those obtained on* a comparable group -of non-FoHow-Through 
children. 

The maj<jrity of the funds was devoted to dafta 'collecting 
and processing fof t'be large number of project locations OE 
wanted included in the evaluation. During the 1969-70 schooP^ 
y6ar, the Institute reported on the analysis of -jthe data it 
had collected .at 53 prospects in school year 196^8.-69 on about 
8,100 children teste*d in the fall and spring^ Inp th45 analysis 
report, dated December 1969, no findings' wereSprovided for^ 
individual sponsors. Due to the developmental jnature of the 
models' at the time 'of data collection and to late administra- • 
ti'oTi of the pretest«/^E decided to f ocuSNa1:tentMx>n orr later 
entering classes^^f cniidr^n, called colioVts, first being 
the group tftaft eritered Follow. Through, in the"^fini of 1969.. 
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, In May 1970 OE convened a panel of experts to review 
the contract foj: the national evaluation. The panel rec- 
ommended that policy questions an^ program objectives be 
identified as clearly as possible. According to the panel: 

"The major concern of this reviewing 'team' 
*. * * lies in the design of the project in terms 
of its actually evaluating Follow Through Projects. 
While the potential of most information collected 
for this purpose (evaluation) was explained , there 
seems' to be an absence of a definite detailed plan 
. designed to serve the major objective of evaluation. 
Imfact, there seems to be some hesitancy to 
evBluate — in either a comparative sense — that is €o * 
compare project against project or to evaluate 
against specific goals." 

OE approved the Institute's proposal for the third 
year of the evaluation on June 29, 1970, raising the total 
cost of the evaluation to about $7 million. According to 
th'^is proposal, the Institute's main task was to develop in- 
formation useful in judging the extent to which projects 
were producing change in the students and communities. To 
accomplish this task, the Institute* planned to test,- 4>^r ing ^/i!^ 
the fall and »spr^ing of school year 1970-71, about 61,400 
children at 107 projects representing 21 sponsors. The In- 
stitute's proposal* stated that comparisons among model ap- ' 
proaches must be made, but^ not until each approach had been^ 
operating at least 2 year's in the m^anner the sponsor intended, 

. / During school year 1970-71, OE recognized the need for 
greater specificity in the ^ evaluation.' As part of a previous 
and^ separate review of OE's administration of study a^nd 
evfaluation contracts^ 1/ we requested .OE *to examine the In- 

»stitute's performance under the contract* An 'OE cffficial and 
A consultant performed the work, 'and OE wrote us on Novem- 

(ber 4, 1970, that: 

***** in the absence of any detailed statement of 
work * *. *, it will be' impossible to determine whether 
the contractor is actually doing the job whi<2h he 
is^supposed to do. .Apparently* several millions of , 

^ \ 

* 

. l./The results of this; review are contained in our r^eport to 
.the Congress entitled ^"Need for ^mproving the Administra- 
tion bf ' Stucjy^nd Evalua.tion Contracts, 8-164031(1), 
'August 16, ;97r.'' ^» • • * ' ^ 
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dollars of effort has been contracted for without 
a specification of the tasks and products of that 
effort," 

On June 25, 1971, the Institute submitted its proposal 
for* school year 1971-72 • On June 29 OE approved the proposal-; 
with 'the provision that the Institute l,ater submit detailed 
plans for the work to be performed during the school year • 
The Institute submitted the plans in September, but, follow- 
ing a r^eview by a panel of experts, OE requested that the 
plans be revised. The Institute submitted the revised plans 
in November but OE also rejected these^^^ " 

According to the Institute, its problem in developing 
detailed plans acceptable to OE occurred because the groups 
within HEW which had responsibility for or direct interest 
in this evaluation contract were reformulating the'pr^^gram 
and transmitting these changes to the Institute. This 
situation was further complicated by OE personnel and 
organization ^changes. The Institute said that during' 
June 1971* the .Follow 'Through Director left the Government 
and a new director wa& named. Also, in July the. project 
officer for the follow Through evaluation and the OB office 
that had cognizance over the proje^:t were changed. 

In December OE decided to reorganize the evaluation. 
In a, December 10, 1971, memorandum to OS's Office for Program 
Planning and Evaluation, the OE official in charge of the 
evaluation said: 

^- "While the responsibility for design was shared with 

SRI (the Institute) in the past, criticisms about the 
overall conceptualization fr-om the GAO and othet 
* sources made it clear that this * joint, venture' mode 
was not a, workable arrangement. Therefore^ thi$ 
responsibility has now been assumed by USOBv. An in- 
tensive study of -tHe overall design of Follow* Through 
is now beihg initiated by USOE.'' 

The memorandum al.so saiSv^that the Institute's latest proposals 
and other evidence indicated that the contractor had limited 
ability to analyze and report* data. 

To 6hange» the evaluation, OE, assisted by a. consulting 
group, assumed full responsibility for specifying the ex- 
perimental design, tests, and 'the projects where testiii^ 
would be done.. In addition, OE selected Another contractor 
to analyze the student acTlievement, teacher, and parent data 
being collected and processed by the Institute, but had the 
Institute continue ^to analyze and repor^t on classroom observ- 
ation data. 
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The evaluation strategy that emerged from these changes 
retained the Follow Through and non-Pollow-Through comparison 
concept* In addition/ rules were estabished ^pverning the 
selection of projects iwhere testing was to be done and which 
children would be tested* Projects were selected to maximize 
sponsor comparisons* ' v . ^ 

According 'to- the retjuest for proposals that OE sent out 
to prospective bidders for th^ analysis f the new evaluation 
was to seek answers to ques-tions such as: • 

— How does Follow Through compare to the^chooling 
disadvantaged children typic^al^Y receive? 

' —What are/fKe benefits of the jjtfdel approaches? 

— Are the effects of the approaches consistent? 

— Have t^ approaches been implemented properly? 

— Does Follow Through have more impact on Children , 
who had preschool experience than on children ' ' • 
* lacking this experience? 

— Do the effects on the chilcjren multiply with each 
passing year? , t * 

— Are the effects of Follow Through »and of individual 
' approaches constant regardless of the child's 

characteristics, such as ethnic origin/ sex, and 

age? 



Thus the evaluation began to address specific evalua- 
tion issues. The cumulative effect of. not having specific 
objectives and ggals is difficult to .assess in^.terihs of 
(1) the costs incurred during the first, 3-1/2 yeard of 
evaluation for testing and other^ activities which will only 
be marginally useful and C2) the extent to which the above 
questions can be answered with confidence. 

Initial definition and scope of program ' 

ft ^ 

Accor<3ing to program documentation, the Follow Through 
experiment was originally designed to provi.de information . 
for use ^when the program was expanded from experimental to 
service or operation4il proportions. However, the specific 
information that OE wanted was not specified to the con- 
tractor until the national evaluation was reorganized 
beginning in the late fall of 1971. 
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data analysi 
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y, the sponsoL^* approaches also were not 1 
indicated in/OE's request for proposal^-^'f or 
which ste^d that: 



ully 



no one [sponsor] was fully prepared to move^ 
into thi primary grades with a completely de- ^ 
velopedL radically different program^ However,. a 
number of approaches seemed to be sufficiently 
well developed and to have enough institutional 
support that including them in Follow Through 
was justified." 

Not until a July 1972 meeting did the OE official J.ncnarge 
of the evaluation request the sponsors to make their/oBjec-- - ^ 
tives explicit and to devise evaluation plans for those ob- 
jectives not covered in the liational evaluation. .^E folloW|ed 
up on this request with a ija^morandum to tlie sporfsors in Novem- 
ber 1972.* ' / ^ ' 




Although the evaluation objectives ajid goals were hot 
precise, Follow Through planners designed a framework for 
large-sca^3.e experimentation and evaluation. From mid- 
December ^1967 to June 1968, OE identified and selected 
14 sponsors and 64 additional praj^cts to participate in 

Follow Through and awarded a contract for ejctensivexaata-- ; 

collection. At the start of the second j^e^ of Follow- 
Through, which, was also the first year of the experiment, 
there were 103 projects and 14 sponsors serving approximately 
15,500 low-income children -in 2 grades. 



According to the .minutes of a March 19^8 meeting which 
OE held to develop plans and procedures fcrr. a national evalua- 
tion, OE envisioned Follow Through an^^iits evaluation as hav- 
ing two^ sta9es — a formative stage in which to develop the 
model approaches and a suramative stage in which to a^&ess 
and compare their eiffects. Some participants at the m'feeting 
suggested that projects still in the formative stage be 
exgjkr^ed from toe* national evaluation, a strategy not closely 
followed in the evaluation that evolved. Although OE and . J 
'^he Institute avoided making sponsor comparisons and the 
Institute collected formative information on the pjrojectjB* 
progress in implementing the model approaches, much summative 
dc^t^ on pupil achievement was also collected which, as 
previously discussed, will not be used for evaluation pur- 
poses or will be only marginally useful. 

OE required the 64 LEAs that implemented Follow Through 
projects in 1968 and encouraged the 39 that implemented /- 
projects in 1967 to Select any 1 of the original 14 apprq^^ches. 
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sponsors were unevenly ai^tribuced araojig projects 



f the country* Some spo 
cts while others worked w 
and^(5ine spon<^rs had th,eir approaches imp 
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"t^e geographical distribution of sponsors b 
ect choices to a list 
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j3ecisions on national evaluatibn. 
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Because test samples used befdre^^^^tl^e^evaluatloji y^s 
reorganized were not designed to asses^^ifeferences ambug 
sponsors, interim findings on the cifrMren who entered th^ 
program in the fall of 1969 and Jr^^jQr give Ivery little in- 
fotifat^ipn about differences amcmg/sppnsor^ — a primary ob- 
""^tive of the reorganized ^^li^ptbn^ 



chart below 
the evaluati^ 



atipftal evaluation involves three 
reli, called cohorts. In' general, 
s, they enter scnool (either kinder- ^ 
de).-;^ at some interined:.ate point and when 
ogr^ at the end of the] third* grade. The 



red 



>ws the progression of children involved in 
through the grades by cohort ancl school) year. 



— Schoor year ( 



note a) 



1^^-70 1570-71 197^1-72 1^72-73 1973-74 1^74-75 



Cohort 1 



Cohort 2 



Cohort 3 



K 

1 



1 

K 

1 



2 
3 



K 



a/Each cohort includes two groups of chi 
entered kindergarten and one which e 
grade. , . . " 



2 

ren^-one which 
erjsd'the first 



1 



\ "According to OE, the data coilected before the revision 
of the evaluation design (cohoc/t^s one and two) will be used, 
but because the data is not a^ stable, nor as large or com- 
prehensive, as the data forycohort three, 'conblusions about 
model, effect iyene^Jis; have b^n delayed. The program's ultimate 
ef fectiveifess will-^be de^^ep!^ined by the decree to which it 
has fostere^^he devel0pipent of successful sponsor approaches 
for teaching yoUng dtsa<!vanfeag^d-.xUlildrent 
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According to an OE official, it takes about 1-1/2 years 
to analyze and report the r^esults ^pf_the Follow Through 
evaluation data. An assessment of program effects should, 
therefore, be available for each cohort either after 4-1/2 
oj^ 5-1/2 years, depending 6n when the children entered 
Follow through; that is, a^ kindergarten or first grade. 

Data on the children who entered Follow Through in the 
fafl of^ 1969 (cohort one) jhas been completely'-analyzed . 
According to OE, sponsor aomparisons have been made, even 
th^ugh.^^in many cases there/ were too few sponsors with com- 
pa^able^rojects where sufficient testing had .been done: 
to permit adequate compat^lsons . 

Data on the children',. who entered the program m the ^ 
fall of 1970 (.cohort two')! has not beeri analyzed , but OE ^ 
plans to use it to compdiie the effectiveness of various 
sponsor approaches at the end of the third grade. According 
to.OE's September 1974 request for proposals to analyze, 
selective Follow Through^data collected throi|2[h ^the spring 
of 1974, data on both the children who entered 'the program' 
in 1969 and those who entered in 1970 (^cohorts oni and two) 
is considered developmental and does hot represent a strong 
analytic data set because of limited testing at intermediate 
points. 

For judging model effectiveness-, OfiThas, th^efoTe, 
decided-^ rely most heavily ^n cohort three'?' Jttiat," is , the 
children who^ntered Follow Through in the. fall of 1971. 
Because the data-^llected on the children en tering in 1971 
is (Bore reliable tFiar^that collected in 1969 and 1970, i,t 
is to, be useKS to answer^asiof the evaluation issues under 
^study! ^6e hopes that^ the efl^x>t5ob served from the earlier 
test samples will provide some indication of the corvsistency 
of sponsor effects. . T*""^^^ 

Our^ analysis, however, as shown in the following table, 
indicates 'that: OE's ability to determine the e^xtent that 
cohort one results confirm those from cohort three^will be 
limited. Based on the 10 sponsors included in. the fir-st 
evaluation report on cohort three and their pro jects^ that 
are s,cheduled for exit level testing at the fend of the 
third grade, cohort continuity is as follows!^ 




Number of projects across cohorts 



Sponsor 



^4 



1 
2 
3 
* 4 
5 
6 

7 • 

8 . 

9 

10 



Total 



Cohorts 


Cohorts 


Cohorts 


one 


two 


one*, two, 


and 


and 


and three 


three 


three 


(note a) 


I 


5 ♦ 


1 




3 




3 


4 


3 


2 


3 


2 


3 


6 


3 


1 


3 


1 


1 


^3 


1 








1 


3 


1 




_3 


J. 


13 


33 






1 1 II 0 * 





a/The projects included in this column were also included 

^ in the count for the other two columns on cohort continuity. 

With regard to cohorts one and two results confirming 
those from cohort three, nine sponsors have more than two ^ 
projects, in cohorts two and three, but, iti cohorts one and 
three, only two have more than two projects^ Also for the 
three cohorts involved in the evaluation, only two sponsors 
have more than two projects with which to determine the con- 
sistency of sponsor effects acrpss adl three evaluation 
.cohorts* The remaining projects scfi'eduled 'for exit level 
testing in cohort one (five) and cohort^two (nine) may still 
be useful, according to OE, to provide some indication of the 
consistency of sponsor effects. At the completion of the 
evalua,tion, however, OE's ability to determine the extent 
that cohort one results confirm those fijom cohort three will 
still be limited, y^^s will its ability to determine trie ex- 
tent that the eval^iiatipn results repeat themselves across 
the three cohorts. ^ 

In summary, cohort one data combined with the data 
collected on the children who entered Follow Through in 
school year 1968-69 will make a considerable amount of 
evaluation data only marginally useful because it was col-' 
lected before specific' evaluation objectives were specified. 

, The former Follow Through Dir^tor 'said in an article on 
the accomplighments and lessbrts learned frfom Follow Through 
that, during, future experiments, program^ impact d^ta should ' 

' not be collected until the program's projects are solidly ^ 
established. 
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Further, as discussed later in this report, there are 
also design and measurement problems connected with the 
experime'^nt whi'ch 'affect the reliability of the Follow 
yl'hrough results, including those for the children who 
entered 'in the fall of 1971 (cohort three). 

OE also plans to assess sponsor consistency across 
cohorts in another way. It will look at the degree to 
which a sponsor achieves the same patterns o'f effects over . ' 
cohorts. For example, a model may have different effects 
on children who have had preschool and those who have not, , 
or in large urban areas as opposed to rural areas. If such 
patterns occur, they will be verified 'on- more than one 
cohort j:o determine if they are true patterns. Tf such true 
patterns are identified, this could be another way of assess- 
ing sponsor consistency. However, the validity of this 
apfHi^ach remains to be seen. ^ ' 

As of July 1974 the contracted cost of the national 
evaluation was about $23.1 million. This amount represents 
the cost of the contracts with the Stanford Research In- 
stitute, the consulting firm assisting OE, and the contractor 
performing the data analysis. iJowever , the total Federal 
cost of evaluation i& even greater because (1) LEAs have been 
paying with grant funds the salaries of assistant testers and 
test aXdes since .the spring of 1970, (2) OE has "made grants 
to sponsors to make separate evaluations of their models to 
supplement the national evaluation results, and (3) OE has 
awarded numerous smaller research and evaluation contract^ 
for various purposes. 

Impact of change from service ^ ♦ 

to experimental program 

Another lQsson:from Follow Through involves the rela- 
tionship between Follow Through as a service and^ as an ex- 
perimental program. The former Follow Through Director said 
in his article that, because this relationship was nevet fully 
established, progress in the experiment Was' impeded. Also, 
a Follow Through review panel reported in March 1973 that: 

"^^* * by congressional authot ization Follow 
"/Through is a community action and* social service 
>A program, while by executive direction it is a - " 

"^^--^ planned variation experiment. This ambiguity 

^Ver the purpose of follow Through has continued 
^.-^to exist in marty people's mindSf both in and outside 
the government, since its earliest beginnings and 
is a key reason Why^ many of the evaluation issues 
have never been resolved successfully." 
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The ,M^rmer Director's statement and the views of the panel 
impljffthat experimental programs should b6 clearly author- 
izedl.as such before they are undertaken* 

EXp fcMENTAL PROGRAM RESULTS * / - 

TO determine the results of the Polaow Through experi- 
ment,^ we reviewed the two available Follow Through evalua- 
tion studies which include informatioa on eduational /gains , 
asilneasured by achjLevement tests* According to the .6e of- 
ficial in charge of the Experiment, those two' studies plus 
one in draft form and three based on classroom obs^ervations 
are the most important studies on the experiment. 

The OE official said that the first two classroom • 
^observcition studies are considered developmental, but the 
third one reports some significant findings regarding class- 
room instructional practices. The results, which are base^ 
,on systematic observations of the Follow Through classrooms 
in action, indicate that highly controlled classroom environ! 
ments contribute to higher ^scores in math and reading and 
tljat flexible classroom environments contribute to desired 
child behavior such as independence, lower absence -rates, 
and higher scores on a test of nonveroal reasoning v'; 



This section addresses the two studies b€sed jart^nieve 
ment tests because such tests serve as the "primary^ nfeasur^ag 
.device in ^Follow Through and most federally fun4^d experi-^^* 
mental education programs.. The focus- is on the more recent 
study because it deals with cohort three — the best one for 
determining model effectiveness. , ' 

The initial results on cohort 3 covered 10 sponsors 
and were reported on March 1, 1974. 'These results represent 
the first from the revised evaluation strategy and reflect; 
the first year progress of kindergarten children whp Entered' 
Follow. Through .in the fall of 1971. -OE, .in a June 1974 / 
summary of that reportl concluded that there is reliable/ • 
evidence ' that systematic dif f erences *among the 10 jspojisdrs' 



the 



most 



)proaches have been acrtrteved. OE concluded also 
the sponsors are showing evidence of developinox^he 
ihildiren's motivation and that. six iare having sprtie effect 
ipoh the chiJLdren's sense of personal responaioility. OE 
)ased 4ts conclusions solely on educationaj^ significant 
(differences between FollowjThrough and nofv- Follow-Through r 
students on eight criterion measures ^/^oup of the eight 
measures tested academic skills an^^^four tested nonacademic 
attributes. 
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"•^/Our^ conclusions on the March 1974 report differ from ^ 
because, before applying the OE educational signifi-^ 
c^ce test, we first determined whetherv a*' difference between 

le gains of Follow Through and non-Folrow-Through children 
iras statistically significant. A -tes^t for statistical sid(- ^/ 
nificance is e^ssential to provide confidence that the differ- 

rs'^^rived are not due to chance .and thaty the resultis can 
be generalized 'to a larger population of interest (e.g.v ^, 
children from low-income f^imilies previously enrolled in ^ 
Head start .or similar preschool programs). OE officials 
saic^'lihey did nqt report on tests of significance in th;^ 
j|ine/l974 summary because the assumptions required for this 
test; to be v^^l'id v^re not met; that is, the VEhs in Follow 
Ttyr^ough wer^ neith^ rap((3omly selected from the population 
of inter^t nor were they rarijfomly assigned /to a sponsor* 
approj^.;/ • y \ / ■ 



Sponsor effects were fneasured in standard deviations; 
that is, a statistical measurement which measures ^he varia- 
tion of individual values from an average value^ OE con- 
sidered a dif f Q^rence^^qual to or greater than^0\25 standard 
/deviation units of. the i^^m being measured as educationally 
significant and ,^^Hrfer e fore , a positive ef^^t for Follow, 
Through. The OE summary of the March 1^974 results and our 
analysis of those results are shown ip^1:he table on page 18. 
Both analyses shov^ the results the Iv sponsors, achieved in 
4 categories: CD academic achievement, (2) achievement " 
motivation, (3). personal responsibility, and (4) attendance. 

When a statistical significance test is applied l^efore 
determining educational significance, the extent of system- 
, atJ-C difference among sponsors is not nearly as great. For 
two categories — personal responsibility and attendance — , 
the ^mo'st per\J^asive result'^ is that there is no significant 
^difference in the outcome of Follow Through and ' non-Follow- 
Through children. 

Because projects, classrooms, teachers, and children 
were selected j udgmentally , the March 1974 evaluation re- 
sults analyzed by us do not have the statistical reliability . 
that is desired for generalizing from the results of an ex- 
periment such as Follow Through. Nevertheless, our analysis 
dqes show that relying on educational significance alone 
could lead to generalizations that overstate the positive, 
effects of Follow Through. OE of.ficials told us that they 
are working with the data analysis contractor to develop 
methods of analyzing the data so*that generalizations will 
be possible. 
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/ Number of sponsors 
QE analysis ^ GAP analysis 



1 
2 
1 

1 
1 
1 



10 



8 
10 



1 
5 

.2 

10 



3 
10 






1 

Ji 

10 



Criterion measures used 

-Academic achievement — 4 measures: 

FT better than NFT on all 4 tests 
FT better than NFT on 3 tests 
FT better than NFT on 2 tests 
FT better than NFT on 1 test 
FT better than NFT on 1 test and 

NFT better, tkuin FT on latest 
NFT better than FT on 1 test , 
NFT better than FT on 2 tests 
NFT better than FT on 3 tests 
No difference between *FT and 'NFT i 



Achievement motivation:' * 
FT better than NFT 

^FT betters than FT 

No difference between FT and NFT 



Sense of personal, responsibility — 
2 measures: ^ 

FT better tnan NFT on both tests 
^FT* better than NFT on 1 test 
" NFT better than *PT on 1 test 
NO difference between FT and NFT 



Attendaijce: 

FT better tl^^n NFT ' . • - 

pio diffe^rence between FT and NFT 



FT:. 
NFT: 



Follow Through.^ 
Non-Follo^i^-IPh though . 



Some in4;erim results op parlier cohor 
were repor.ted in February 'l973 and cover. 9 
analyzed in the March 1974^ report. Those 
did not shcJW the degree o£ sponsor diversi 
the March 1974 report* Althoagh. some' smal 
was apparent, none of the sponsors had the 
or failure shown in the March report* Mpr 
that showed strong gains or' losses In the 
showed completely* opposite results in , the 



ts of the ex 
of the 10 s 
results gene 
ty indicated 
1 degree of 
widespread 
eover , three 
March- 1974 r 



earlier one'* 

se'quently^'it is'difflli'ult to ^sta^te* that sponsor trend 



been established* 
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Because only interim results are available for cohorts 
1 and 2 and^cause those for cohort 3 represent achi^eveinent 
at the end oqi^nly 1 year of parjticipation , ^ it is too early 
to draw firm conclusions about th^ stability of the results - 
of the 10 sponsor approaches analyzed* Some sponsors may 
not be able to maintain the gains shown to date, whereas 
others njay not produce gains until the later years* of the 
experiment. Further # any significant effects from the 

pproaches may not be immediately apparent but rather may 
occur in the first', second, or third grades or even iatec . 



OE has no formal plans to evaluate the progress of» 
^-t^hiidren after they graduate from Follow Through. In 
'M^rch 1973 a Follow Through review panel recommended such 
an evaluation, but a decision to follow up on Follow Through 
children has not been made. OE is, however, considering the 
feasibility, of testing former Follo*^ Through and non-Follow- 
Through children entering the fourth grade in school year 
1975-76. 

Reliability of 'exper imental results 

Ev/fen though OE reorganized the -evaluation design for 
Follow Thorough during the fourth year of the ,ej|^riment; 
problems in both the initial design and implel^fftation of 
the ex,per iment' limit attempts for overall conclusions that 
a,re statistically reliable for evaluating alternative ap- 
'proaches to early childhood ei3uchtion. These problems include 

' — no random assignment of LEAs to sponsors and . 

' — differences in initial achievement levels and socio- 
economic character istics between the experimental 
* ^ and control groups., ^ ^ ^ \ 

lTr^aditi9n,' the data analysi? contractor listed in the 
March 1974 report seyferal reservations concerning design and 
measurement problems which also affect the reliability of 
results. Two such reservations were (1) methpdolpgical 
limitations stemming from the^lacJc of available knowledge 
in modern education research to measure .all the Fallow Through 
variables and« (2) inability to identify the "best*^ sponsor ^e 
to qualitative differences in materials an^ ''procedures .usQ<r^ . 
and in .functions and skills emphasized. The reservations 
combined with the problems, which We believe cannot practi-' 
cably be overcome, r^aise questions- about the dependability of 
the evaluation to Judge the approaches. ^ \ 



Because LEAs were not randomly assigned to an approach 
and because most were not restricted in their choices of 
approaches, the children in Follow Through and their com- 
parison counterparts are not representative, of any definable., 
larger populations and the overall distribution of pfojects 
among sponsors is irregular. The significapce^of this is 
that generalizations cannot be made beyond the' character- 
istics of the groups of children', parents, teachers, or . 
institutions analyzed. AccordiHig to the data analysis con- 
'tractor, ethnicity, integration, city size', and geographical 
region are all associated with student effects in some way 
or other, but nbne^ are uniformly distributed among sponsors. 
Also, because LEAs were, for the most, par t ,. permitted to 
select sponsbr^s, the contractor, could not estimate whether, 
the outcome of one sponsor's program would be similar ^to 
another* sponsor or whether other p-rojects would respond 
similarly to a particular sponsor. 

Another result of most LEAs not being restricted in , ' 
their choice of approaches is that someXsponsor s were 
associated with only a 'few projects. Th| evaluat<^ion design 
for cohort three — the one^ OE plans'to rerymost heavily on 
to determine model effectiveness — requires th^t a sponsor be 
working with at least .five projects where adequate testing' ' 
had been done to be compared with other sponsors. 

Only 7 of the 22 sponsors met^that requireraeat < In 
addition, the March 1974' report includes three others, two - 
with three- pro jacts* and a'third one wit'h. f^^J^ projects. 
According to thj5 OE official in charge' d£^ the revaluation , 
five projects are needed per sponsor • provide^a confidence 
level which insures that Follow Through effects are attribut 
able to sponsof intervention rather than other factors which 
^an affect educational achievement. - > • , 

Differences^' in initial achievement levels and kocio^- 
economic characteristics also affect the reliability of the 
evaluation re'sults. Because^ Follow Through and non-Follow-* 
Through children were"" not randomly selected for participa- 
.tiqn, in the> experiment, the experimental and control groups 
were pot -comparable in terms of initial achievement and. 
socioeconomic characteristics. In an effort to adjust for 
the initial differ^ces, the Sata analysis Contractor used 
a statistical technique known as the'analysis of .coyar iance. 
feowever, the contractor reported that the Follow Dhtough , 
.data failed to^raeet^some requirements believed necessary , 
for this technique ^o be an effective adjustment device. 



Further, no known statistical 'technique can fully 
compe'ns^e for initial differences" cm such items as pretest^., 
scores and xSocioeconomic ^character istics. Accordingly, as 
OE states in its June 1-974 summary, "* * *the basis for- ' 
determining the effects of vario^u^o How Through models is 
not perfect. " Our review of*the March 1974 report indicated 
that, for' at least four sponsors, the adjustments were 
rather extensive. Included among the four is the only sponsor 
that produced significant differences on all four academic 
measures and the only two sponsors that produced any academ'ic 
results significantly" below their rtonTFoHow-Through counter- 
parts. ' ' ' . ■ . 

According to OB, while it is proper 'to observe that the 
lack of raiidom, assignUehfe of ..treatments and the lack, in' 
several cases", of. comparable Control groups will make it 
difficult to draw conclusions about the effects of -Follow 
Through approaches, ,suc'h conditions typify social prog'ram 
evaluations and educational evaluations in particular. We 
recognize that these prO'blenis afe not peculiar to the 
Follow Throilgh. program, but believe that, when the decision . 
was made to conduct' the program- as an- experiments -greater 
care could have .been' exercised to, minimize the effects of 
not following the ideal experimental procedures." 

~ ■ * ' ■ - ■ ^ , 

Because there is no known method as yet to fully com- - 
t>ensate for initial differences between experimental and 
Oontrol groups and because many previous educational ■ experi- 
ments have tended to indicate relatively small gains, the , 
more -the ideal procedures ca'n be implemented, the leas • 
susceptible final results will- be to serious mismatches ih 
the two' groups. 

Combined with the shortcomings in. the experiment ' s 
design: and iraplenleJitation , the lack of complete i^ta m the 
data collection process also affects the .representativeness 
of the ev-aluation results* Our review. of. the contractor s 
data in the March 197-4 repcjrt plus source documents desGrib- 
'ing the national evaluation indicated that-* fQr^hp 10 ap- 
proaches covered in'the March report, only 48 of the 93 proj- 
•ects that:*raake up those approaches were included. ; The , 
projects not inol^ided were, for the most part, ^hos'e that, 
lacked suf f icientSaseline data. Thus,, it is .questionable 
whether the p ro'jects-se ported On are representatjve overall 
of the— i^'^Sppro aches* . . . ■■ 

For the 10 spon§ots,. the. data analysis 'contractor 
indicated further limitations in that -the results analyzed 
were for those children for whom data on all thfe ^important 



inea'sures could be obtained* For 'example, of the 6,0Q0 Follow 
^hrough children tested for <:ohort 3 in, the spriiig of 1972, 
J^,,000 were jiot reported on in the ^March 1974 report. About 
40 percent of the' 2,0,00 .children .not reported on were lost 
Jto the program through attrition*' 

*0E estimates that about 50 percent of J:he kindergarten 
children' for ^cohort" three will be Jost tbrc^h attrition 
by the end oOBr^^dg.,^thr£e. The consequence""&^-Nalttrfition 
on the Follow Thrbu^h experiment is that the -remaining 
children may nit b^Trepresentative of all of the effects of 
the program. ^^^8^^ result , there is no assurance that a 
.representai^iv^number of students has been included in the 
analysis. » i * - 

PROGRAM REPLICATION , 

« 

• * During hearings on OE's fisc-al year 1974 appropriation, 
OE officials to-|d the Congress about the planned termination 
of Follow Through and about' their ultimate intention to dis- - 

.seminate successful Follow Through models to LEAs throughout" 
the Na-tion, if the final results of the national evaluq^tion 
warranted dissemination. Because OE still plans such dis-- 

fSemjination ^nd has developed tentative success criteria to 
judge the rarodels, this section will address the need to de- 
velop 'advance plans to accoittplish' this. 

/' . The -iJttplementa,tioa process is important. b.eca^se it - - 
rtust provide reasonable assurance that the result's achieved 
T)y the experiment will be replicated (repeated) in the new 
communities^ ' Thus, the p^roces? OE s^elects should fee tested 
before it is used for widespread dissentination of ^llow 
Through models^ ' ' ^ , ^\ ^ ■ . . 

OB' .-has- not ^yet developed appropriate plans for refpli- 
cating Foitiow Through approaches.' During the fall of 1971 * 
OE^ had started a long-range plan to expand th^" number of 
Follow Through^ projects in each State but terminated, it. 
al3put 6 month^'s later. Afe of April 1974, OB'-s Office of 
Planning, Budgeting, and /^valuation' was implementing a new 
concept known as "Pirtoject Infarmation Packages" for replicat- 
ing successful compensatory .education projects, but OE of- 
ficials cannot agree on whether " Follow Thr6ugh models should 
be r.eplicated under 'this concept* 

Xongi»rangfe plan . 

, / • . - : ' ' ' . ^ 

O&^s goal for -this plan had been to .develop within each 
State over a 5-year per ipd thg capability to bring Follow 
Through to as many children from low-income . families *as 



possible. Besides anticipated increases in the Follow 
Through appropriations for thp additional prbjectSr title 
I of the Elementary and Secondary Education Act and State 
and local funds were to be used. 

OE planned to select existing exemplary projects and 
tprn them into resource centers for new project staffs to 
contact or visit for information. In addition, OE planned 
to retain the sponsors to work with and help the new proj- 
ects implement successful^ approaches. 

i ' ' ' ^ ^ 

In short, the long-range plan wcful<3 have involved {1} 
i gradual phase in of the Follow Thz;ough experiment, (2) 4in 
increase ir\ the number of communities working with Follow, 
Through models, (3) the conversion of Foilow Through from* 
a research and development to a predominantly service- 
> oriented program,' and {4*) the merger of the Follow Through 
concept into the operation of the title I' program. Greater 
support for Follow Through from title I was planned because, 
according to OE, "components of a typical Follow Through 
are [the] same as an effective title I project." 

In October 1972 an OE official told -the Follow Through . 
program dffice^that its 5-year replication program had been 
rejJlaced by a new incentive grant program being developed ^ 
for title I. Later, OE^announced its pians-to terminate 
Follow Through. 2in'0E planning official tald us that the 
decision to replace the 'long-range plan had been made in 
the spring of 1972 and had been based' on a planning philoso- 
phy that research and development p'rograms, such as follow 
Through, should not have service objectives. He said that 
Follow Through models should not^be replicated until all 
the results are in and they have proven successful.. Accord- 
ing to the official, the incentive gran-t program was envi- 
sioned primarily as a program for replicating successful 
reading and mathematics projects but the Offifce o^ Managifnent 
and Budget did ,not approve it. 

* Project information packages 

*The project information package concept involves col- 
lecting and assembling information needed to install an 
. 'educational approach in the classroom.* -ft package will con- 
. tain descriptive literature, schedules, budgets, procedural 
guides , f ilmstr ips and audiocassettes , self- instructional 
training programs ^or teachers , lists ,of teaching aids and 
equipment, and other materials needed to install and admin- 
, istet an approach. OE hopes that the- packages will be 
"Sufficiently self-explanatory to enable school administrators 

Si •■ . • 
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to install an approach and achieve successful results 
^ithodt ?nuch assistance from the -sponsor. 

, To determine .whether the concept will work/ OE .has 
identified six successful compensatory education approaches 
^(nOne of v^ich were Follow Through) and collected t;he 
necessary information to package them. Each packaged approach 
is .now being field tested a*t several LEAs. 

According to officials in OE's Office of Planning, x 
Budgeting, and Evaluation, Follow Through approaches are 
candidates for replication undej the packaging concept. 
Howe^ver , because the' packaging concept does not necessarily 
provide for the input of the sponsor at the replication sit;e, 
the Follow Through Director believes that^ Follow Through ap- 
proaches cannot be successfully impremented under .this con- 
cept.. She favors a program more like. that provided for 
under the ,iong-range plan. However, she. said that OE has 
not yet developed appropriate plan^ for replicating ^Follow 
Through approaches. ' \ " ' ^ 

In planning for replication, OE should coris^Jer , among 
other things, the n-ecessary cost. According tc^-a 1973 cost ' 
study done under contract for Oe, the average Federal per 
pupil cost of Follow Through in school year 19.71-72 was 
$747. This cost is in addition to the LEAs' normal per 
, pupil expenditure. The 41 projects studied wer^e believed 
to be representative of the 173 in operation -dllEing school 
year 1971-72. The $747, in addition to Foll'ow Through 
funds, includes title I of the Elementary and Secondary 
Education Act and other Federal funds, sucft' as model cities 
and manpower training., „ « . ' 

During OE^s fiscal year 1974 appropriation hearings, 
OE of f icials^^ld the Congress that in school year 1972-73 
Follow Through ' s ' aver ^ge per pupil cost had dropped to 
$617 and tnat its -goal was to lower the cost to $600. In 
Jtomparison, t*he title I program, similar in services to 
Follow Through, cost only about $220 per pupil for school 
yeaif 1971-72, the most recent for which OB statistics were 
available. ' ' ^ 

<:ONCLUSIOb^S 

School /ear 1973-74- represents Follow Through^s seventh., 
year of existence and its sixth year of experimentation with 
approaches to educating disadvaataged children inrtne early ^ 
grades of school, Throiugh* June 1974 an^estimated $300 mil-^ 
lion^had n^een ^ade available for the pr/ograftt, and according 



to OE abo-ut 71 ,000 children ^had completed it. Problems, 
flowever, in both the initial design and implementation of 
the experiment will limit OE^s ability to reach- statistically 
reliable overall conclusions on the success or lack of 
success of the approaches for teaching young disadvantagea 
children. »^ 

The oroblems cannot practicably be overcome, and, wlien 
combined with the OE contractor ' s reservations about des^ign 
and measurement problems, raise questions about the ^ei^per i- 
ment's dependability to judge the appproaches. Incomplete 
data in the data collection process also limits the 
representativeness of the evaluation results. The problem 
of data reliability could have been minimized if OE had, 
when Follow Through' s emphasis changed 'from service to 
experimental, developed detailed specifications for the 
evaluation. 

.On the basis of 10 sponsor approaches, initial ^i^ults 
from the revised evaluation indicate that there are ^Ttfer- 
ential effects among the approaches. It is too early, how- 
ever, to draw conclusions about their relative merits. Be- 
cause the evaluation data collected during the first 3 years 
of the experiment is of questionable usefulness for judging 
model effectiveness, OE has to rely most heavily on data 
from the' last group of children participating. In addition, 
OE will have only limited opportunities to determine the 
extent that the evaluati^ re^ult^ repeat themselves across 
certain cohorts. ' ^ 

The questionable benefit of the Follow Through experi- 
ment affects OE?s plans to disseminate information on 
successful approaches. Bepause limitations in the data 
affect OE's ability to draw reliable conclusions about the 
success of the approaches, such limitations also affect 
OB*s plan.s for replicating approaches. 

In the future, experiments generally should be designed 
on a small scale and 'have a limited number of projects t^ " 
define implementation pr.ocpdures and program objectives and 
.goals. After development, the program can be expanded to 
more projects, if necessary, to assess its ef fects . r f ther 
any evaluat^ion done during development should be/ on a small 
scale, as an aid to developing the program- rather- than as- 
sessing its effects, as happened in Fqlldw Through: Irt 
future large-scale experiments, the framework for 'th-e ex- 
periment ' should be designed at the same time the evaluation- 
objectives are established so that one wrH not limit the 
effectiveness of the other, as happened in 'Follow Through. 



OE Should use the Follow Through experinfent as a case 
study to plan and carry. out future exper imefital programs. 

RECOMMENDATIONS TO THE ' / 

SECRETARY OF" HEW ' 

We recommend that the Secretary direct OE to: 

. —Develop ^ea^^ er safeguards to insure that future 
experimental programs are not designed apart from 
evaluation to maximize the degree to which experi- 
, , mental res^qlts will be statistically reliable. 

— Translate .the problems encountered during the 
ipdllow Through experiment into criteria for design- 
ing and administering future experiments, i^^cluding, 
among other-; things , criteria to' prevent collecting 
massive data on program results before program goals' 
and objectives are properly defined. " 

AGENCY COMMENTS 

, HEW, commented on matters discussed in Ur^s report by 
a July 28, 1975, letter. '(See app. I . J^J^concurred with 
our recommendations and said it will/^y to insure that 
legislation and. regulations for futute experimental pro- 
grams are consonant with strong evaluation designs by 
requiring random assignment of projects to LEAs, schools, 
classrooms, etc., and requiring LEAs to establish and 
maintain comparable control groups or schools. OE will 
formally advise ail program offices of the importance of 
faking evaluation an integral part of experimental pro- 
grams. 

HE;w said the Experience of Follow Through reemphasizes 
the need for planning before implementation of new 'program 
designs, or majpr changes in basic optogram thrust. ' OE now 
has a' centralized .evaluation staff responsible for develop- 
ing evaluation designs based upon , program objectives- so as 
to prevent unnecessary tiata collection.. HEW said this staff 
IS cognizant of the problems encou^te'red in Follow Through 
^and will, to the extent possible; incorporate ®ur recom- 
mendations in ttase overall evaluation designs. 



CHAPTER 3 y*-- . ^.^ 

tJE ED TO IMPROVE PROJECT ADMINISTRATIOl/ 

AND PROGRAM MONITORING 

Follow Through offers, vacious noninstructional services 
and activities aimed at lessening the effects of poverty and 
enhancing children's chances to lead normal, productive 
lives. These components consist primarily of parent, involve^- 
ment; community involvement; and supportive Services, such aS 
medical emd, dental examinations for the children. 

In general, the LEAs had provided for each required 
service and activity but needed improvement in their admin- 
istration. Project achievements in thfe noninstructional area 
varied from one LEA to the next and a nee_d existed to in- 
\crease the end results. In addition, LEAs/ needed to improve 
\their recordkeeping for evaluation purposes and, i^n some, in- 
stances, to comply morre closely with funding requirements. 
\ {• ■ . ■ ^ 

The reasons for these weaknesses and the uneven results 
among LEAa in delivering services seemed to be that(l) OE 
had not furnished adequate guidance to implement the non- 
instructional components and (2) OE's monitoring of project 
results had been ineffective. During our review OE improved 
its monitoring system, but additional changes are' still 
needed. ^ 

PROJECT. ADMINISTRATION OF SERV-ICE . . 

COMPONENTS AND .^RqgRAM REQUIREMENTS 
. NEEDED IMPROVEM^T 

We examined the LEAs* ♦performance in various areas, in- 
cluding the delivery of medical, dental, nutritional', guid- 
ance, and psychological services to eligible children and ' 
the delivery of social services to both el igibi^e children.. 
^' and their families. In addition, we evaluated the extent 
•and ways in which parents and" other individuals and agencies 
from the community were involved in pro ject affairs, includ- 
ihg coordination'^^with local Head ,St^rt projects. 

Although each project differed in many ways, such 'as 
geographical.* location and project >ize, all needed to im- 
prove.,-t>roject administration to some extent^ Services could 
have been, increased, recordkeeping needed improvement to en- 
hat\ce evaluation", and. funding reqi^irfemerits were not always 
complied with. -In addition, the-LEAa varied in the level of 
serviqes provided. Highlights from selected areas are dis- 
cussed in the following sections. . ' ^ 



Variation in and need to increase services 



The draft program manual, which was in effect at the • 
time of our sitewo'rk, contained guidance to LEAs on the 
ij;equirements for each program component. It provided that 
each project he^ve a policy advisory committee, made, up of 
^ parents and other community members .whp must play^ a sub- 
' stantial- role in planning and managing ptoject affairs., 

* . - 

Of the^nine projects, seven had a ''policy advisory commit- 
tee, one had a parent board, and one had two parent boards. 
The two projects with parent bOards ^^e par^^nf* implemented 
in that the parents, through the board^onade' the program's 
operating decisions. Most committees and boards were orga- 
nized along the lines suggested in the program manual and met 
regularly, usually once a month. 

To determine the extent of committee and board involve- 
ment, we reviewed the minutes of their meetings; observed 
meetings; and interviewed project officials, committee and 
board iu^mbers, and parents' of 'Follow Through children. In 
our opiiiion, one committee and one board were actively in- 
volved in planning and managing project operations, four had 
a voice in^project deci6)iphmaking but a need for greateif in- 
" vblvement existed, and three had little involvement in- project 
affairs. * ' • ' 

The active committee and ^. board made numerous important 
decisions •and actions. For* -example, they both established 
project staff selection Criteria, organized parent activi- 
ties, and helped develop^the project's annual budget. In 
'addition r thefi^arent baard hired and dismissed 'sta'ff members 
and considered educational matters, such as a decision to 
release teachers from their normal duties to visit |>arents 
. periodically. , \ 

Eour cpmrnittees^made a few such decisions but, could 
have improved their performance. The major emphateis of three 
of these four committees appeared to be on their own, opera- 
tions, including selecting officers, forming subcommittees, 
selecting representakive.s to attend conferences, ano^anning 
socictl gatherings, rathef^. than on program matters, sucS as 
developing plans and crit:eria for obtaining parent .and TOta- 
munity involvement. The effectiveness of the fourth commit- 
tee was questionable because 13 of 21 Follow Through parents 
we interviewed said they were not even aware the cdjranittee 
existed. . - * : 

The, minutes of the three committees which h^id little 
involvement in project affairs showed Tittle evidence {yf>^ 
accQjnplishments. Pro ject .of f icials; ^nd committee members 
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acknowledged the passive role that their'Vcommittees had 
played in ptoject operations. They gave us the following 
reasons ^o^^lack of committee involvement. 

—Lack of interest, experience, and (education of low- 
income parents. . 

— Substantial turnover of committee membejs and offi- 
cers. At one project the parent coord^ator told, lis 
th^t the leadership ability of the committee had been 
afiected because o£ lack of continui^j^v^ committee 
members, ^ ' 

— Reluctance of members to expre^^, themselves in the 
presence of the project directpt:^ who attended commit- 
tee meetings. This occurred '"^t:;-<>nly one t>J^oject. 
The apparent reason for. the memfcfe.l?s. not speaking out 
was that more than one-third ^j^^ev also prpject staff .x 
The committee president believ6ch'that the employer- 
employe-en^latipnship detracted from, member objectiv- 
. ity. Thi^^ommittee was being reorganized at the 
time of our \sitework. 

The projects were taking actions to increase committee in- 
•volvement when we completed our sitework. 

1^' Parent participation in the 
• Follow Through classroom 

The 4raft program manual also' prov^ided that. parents be 
used in the classroom, in the Follow Through office, or in 
other capacities working,4S volunteeirs. To assess project 
performance' in this area?^>Ve examined proj,ect records and 
interviewed parents and project personnel. 

The recorder, although not always complete, showed wide 
variation 'in parent participation among the projects. For 
example, one project which served 256 families recorded in* 
school year 1970-71 -about 14 'days spent by parents in, the 
classroom or* on othfer project, activities. In-contrast, an-;- 
other project which served about 900 families recorded about 
600 days^ spent by garei^s in the classroom. Its officials 
felt that/ev^n^ore pare^nts should have participated , but 
r- said tha^ some o^dnjG^ have transportation. 

In addition to the first project cited above, three 
other projects appeared to haVe little parent involvement in 
the classroom. ^ 




— Of 1J teachers interviewed at one ^project, 4 said that 
pai;^ents- were not involved in cljissroom. activiti/s and 
9 said that parents participated but only to a limited 
extent. ^ 

>- The pjrogram coordinator at.. another project s&id prir 
cipals, at the Follow Thirougl) schools prohibited un/ 
trained persons^ fronhworking in the classroom." T^is 
.project'jB records for school year 1970-71 showedM:hat^ 
only one parent worked as a classroom volunteer. ' . 

^ht the remaining project/ which consisted of 15 Folio 
Through classroo^ns, a consultant reported that the 
teachers at the sTchool where 12V of the' classrooms 
were Ipcated gave visitorjs the jimpressiop they were « 
not welcome. Lat6r, the -jXrojectN^istr ibuted a ques- 
tionnaire to parents. of children \ho attended this 
school.; Forty percent, or 32 of t\e 81 parents re*- 
spcmdirig, irfQicated - that they felt Npnwelcome in the 
school or that the scljool treated tiem as intruders.. . 



Health' .care 
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Each project child from a. low-inccffiie family is eligible 
to receive a medicar\and dental examina;tion with ^elated tests 
ohce every 2 years while in Follow Through. To measure the 
extent to which this require^nent was met, we selected about 
1,^300 children^ from the 9 projects reviewed' and determined 
the number eligible for ex^aminations and of those eligible, 
the number who received' examinations and needed . treatment . 

, Health, recprds showed that 53 percent ,of the approxi- 
mately 880 chiXdcen in our S^ample who were eligible JEor' medi- 
cal examinations and about 80 percent of the approx\imately 
900 children eligible for dental examinations received them. 
Of the eligible chil<3ren examined, 3(7 pejrcent needed medical 
attention and 70 percent needed dental treatment. About 
75 percent of tlie children -so i?ef erred were treated. 

For the following reasons, not all eligible children 
were ^examined: . ' 

— Some project nurses did not,. , understand ' the program 
, guidelines in that they thought that only o,ne ^medical 
and oiie .dental -examination were required while a child 
- * was in Follow Tljrough. ' ' .i. 

— Some parents refused or delayed signi"ng parental- per- 
I, . mission slips wh^h were .required befbrie children 
> - could- receive health services.. /----O^ 

^ —Staff was not always aded?uate. ^ 
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■ ^> The projects whfch achieved the heht eJ^amtnation results 
were those, that obtaihed parental permission early m the 
school year and at' o^e time for .health services con tern- 
plated/ Also, as followup/ one- project . had estab'lished the, 
procedure -qf 'Showing on the individual health records add?.- , 
tibnal services, that each child -needed^ to receive,- . 

PKo'iect records needed- improvement; 

'The dVaft program manual did not specify what records 
projects should keep to document their perf oripcThce . We bfe- 
lieve that guidance is needed because some recordkeeping ^ 
improvements were needed at all of the projects reviewed. 

Documentation supporting the eligibility of the children 
enrolled in 'the program waS incomplete at all but one proj- 
ect. This matter, discussed separately and more fully 
chapter 4, is- introduced here because projects needed com- 
plete eligibility records to adequately administer the non--_^^ 
instructional services paid for wit^ Follow Through funds^^ 
Because only children . from low-income families are entitled 
to the^e services, records were needed to identify the family, 
income of each child enrolled in the program. 

Most ' project's also needed to improve their recotdkeeping 
in the health area, because many- records "were incomplete. 
One project >had kept no "health records before school year • 
1970-71. At another project, of 100 children we selected at 
random to determine whether they had received J'^^^^^al .exami- 
nation, 9a had incomplete or unavailab-le -rfpords. At/a third 
project,, where we also selected at random 100 childrjTn, im- 
munization . records were inadequate for 40. 

Two projects kept heal-th information in more thaVi one 
place. At one of these projects, for example, children s 
•health folders at two of the three schools were kept ^n the 
classrooms by the teachers instead of by the nurse. More- 
over., the folders did not show all the services provided be- 
cause some data was maintained separately in the principal s 
office or in the social worker's files. / 

Five projects rieeded-to improve their . recordkeeping of 
home visits. At pne pro<3e'ct, records were not current. At 
another,, recotds of some hom6 visits were not kept, and those 
on file- did not -clearly show the reasons , for the visits; An- 
other project recorded each home v'isit in a log book. The log 
showed the number home visits that had befen made overall^ > 
but information ab6ut individual families, such as how often 
were they visited; .Why were they visited, and what additional 
visits .needed to be made, was not readily • availably. 



; > - . ^ • ^. , 

In our opinion, good records are essential to evaluaticm 
which is esserifial to good management; accountability;" and, 
in ah ^experimfental 'program, the interpretation of results. 
Without adequate records, areas warranting improvement may 
not be identified and the ability to compare project results 
IS limited. ' . • ' ' 

Administering funding requirements 

All the projects reviewed, Tio some extent, either did 
not meet or could" not proves th£(t they had met program re- 
quirements pertaining to the sa&rce, use,. and Control. of 
funds. , - 

, ■ , • ^ . . ■ ' 

^ ' None o^ the .LEAs could prove that they had contributed 
the non-Federal shares 1/ required byytheir approved project 
applications." lA school year.l970->i these LEAs claimed to 
have made contributions totaling aikiut $525,000, of which 
at least $230,00^ was either inadequately documented or not- 
documented at all. About half of the 5230,000, which was' 
claimed as m-Kmd conttib'utions by one LEA, was based on an 
o overtime account which did not show specific dates, times, 
duties, or signatures of persons receiving payments.. In addi- 
tion, some of the docume'nted contributions .were questionable. 
^-co'^cq)!^"'^^^' project cj.aimed as an in-kind contribution 
§2,590, representing the yalue of admission to a- final dress 
rehearsal of a Qircus that was free to anyone in the communit^ 
.who wanted to attend. . • 

Four L^As had not. contributed the required amount of 
title I funds to their projects in- school'year 1970-71. For 
example, one project's., budget indicated that title I funds 
would be used to procure setfvices, equipment, and materials"' 
valued at about $90,000 fot the project. .Although documenta- 
tion, was generally lacking, the LEA's title I director said 
about $49,000 of the planned co.ntribution to .Follow Through 
had not been made. Another project did not know how much of ' 
- the rexjuired^ title I contributions had been made to Follow 
Through.^ At our -request, the' project computed the amount 
actually contributed and found it to be '$10,880 less than 
vthe amount proposed in the project application. * 



1/To supplement Federal funding, OE required' each grantee "to 
.provide Annually, de'pending on the number of grades served, 
- If to 25 percent of. -the- grant award, coi>sist4ng of either 
cash or- in-kind contributions which are / services , materials,, 
and property owned by. or donated or. loaned without charge 
,,to the pr.oject. Grantees are required to show that con- 
^ . tributions claimed have actually been made.- . 



' Although Follow, Through funds for health services can , 
i>e. spent only for 'children froin low-income famiries, at seven . 
projects some children, from noB;:aow|incbme families received 
,rfcbe«€^e'r.vices at prograro expensV^^fit 'most projects this- 
"occurred 15e'6au§«"^ptoj'ect ■ sttff f-s» prf|ide<3. h«rith .sgrvijje^, ^ 
without making eligibility detfermihations. One project', for •• 
example, spenf aboUt $1 ,480 in .school year 1970-71 for medi- ■ * 
cal and dental Services for 51. o'f the project's 53 children 
who were ineligiSLe" for health-services. We brought this 
situation to the attention of the project director who tooK 
action to'' correct this practice 1 

Significance of weaknesses in administration 
on program ef tec.tivenfe^s . ' 

In our opinion, the .variation in services and -other 
needefd improvemen'ts decreased program effectiveness in t^o 
ways. '. From a service ^^iewpoint/ -some of ^the :pot#ntial oT . 
helping eligible children and their 'families overcame the 
effects of poverty was/ not maximi-zed. Prom an experimental 
viewpoint, the follow T^hrough program ha4 been- designed to 
include the services a^fci -activities becausfr of ^ the belief*, 
that th-ey. would help a child' sjjccjeed in school. Because the 
national evaluation did not include measuring the impact of 
the services*, they should have 'been .-constant among projects 
to th^ extent feasible so that, differences in' educational 
-attainment could not be due in part to differences in the 
level of services. / / 

NEED FOR MORE PROGRAM GUIDANCE " , 

^Before the final Follow Through resulations .wefe issued 
on April 21, 1975, OE' s . project guidance was contained pri- 
marily in the 'draft program,manuair? V^^i spanual set forth , 
the required program cojnponents, sijji^as parent p'artici^a- ' 
tion, and described in general terrmsV services, and ^acfcivi- 
ties desired-. under e^h component* ' ' ^ . ^ 

- The *manu^ con^t^iined .little criteria, defining the de^ 
sired level of pra^ect performance in eajch component. In , 
addition, the manual provided no guidance on recordkeeping to 
document project performance and did not' requii;^ projects to 
formally evaluate their activities. The manual stated that: 

"Although there are no fixed' requirements for ideal .* 

evaluati^on, ajp|)licants may devise procedures to 

study questions of particular local interest in 

their^ 'projects and to receive program feedback." , » 

The final Follow Through regulations,, which became effec- 
tive June 5, 1975, specifically define program component 
objectives, but do not contain specific performance standards. 



They do provide that refunding of a project be based 1 
on ^ow w^ll'the LEA provided the required services a"^ 
ties, • . ' - ■ « . . \ ^ 
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Iqi^^uj^opini^p^ difficult to liold LEA/(of£i- 

Ajals' ^ccouftt^lf: for*' theii; -perform^rf&e uiil««-the ,e?;ti4jvt pf 
required servicell and activities is specified. Therefore^^ 
OE sh^buld, to the- extent' practicable , define the perf (Jrmance 
standards which projects must meet to obtain refunding. Per-* , 
/f6rmance standards 'are needed to identify areas needing im- 
prov^menfc and to establish a higher and more uniform level 
' of performance throughout the program. 

The final regulations ^ipulate that LEAs must comply 
with all evaluation procedures which OE may establish and must 
^ participate, ^ if requested by OE, in any periodic evaluations 
of the Follow Through progi;am. / The regulations do not provide 
for self-evaluafeion which/ in our opinion, is an es&ential 
part of project management and which could help eliminate ^ro- 
5ram weaknesses of the' t/pe* noted in our review. 

• On January 1, 13,73, HEW's Office of Child Development 
issued performance standards to be used in administering and 
carrying out the. Head Start program, which provides the same 
kinds of s^vices and activities as Follow Through. In gen- 
eral the performance st^ndafdS/ .issued* as part of the **Head 
Start Policy Manual," pertain to the methods and processes 
^used by Head Start grantees to meet the needs of children* 
The standards . recognize that programs and approaches should 
. . and do vary from community" to community. However, Head S.tart 
policy requires that, no matter what approacli is followed in 
a local program, the standards constitute the minimum require- 
ments tfiat must be met by the grantee' as a condition of Fed- * 
eral funding. Oh January 31 , 1*975, HEW gave notice in the 
, Federal Regi.ster that it proposed to issue tlje He^d Start 
^ perfornrance standards as regulations.. These regulations , be- . 
came effective on July 1,-1975. 

On February 1, 1975, the Office of Child ^Development also 
issued, as part of the "Head Start Policy Manual,? a self- 
assessmeat/v.alidation * instrument for grantee use in' conduct- 
. ing an annual assessment of their activities to insure that 
the performance standards are being met. The instrument will, 
also be used by the Office of Child Development to validate 
grantee evaluations. ' * ^ • ^ 

OE CHANGSS AND OUR SUGGESTIONS 

FOR "MORE EFFECTIVE MONITORING . - 

. •Xn effective "raonitoring system is one in which pj:ogi:am 
^activities are syst-ematically reviewed to deterjnine whether 



they are being conducted as planrted. . If deficiehcies occur,- 
the system should provide for identifying causes and recom- 
mending solutions, something that^ at the time of our site- 
work, OE's monitoring system did 'not effectively do. A need 
existed to increase project performance and improve project , 
recordkeeping. In addition, all -projegts reviewed, to some 
extent, either^did not meet or could^ot .prove that they had 
met program' r^irem^nt-S 'pertainip to\th^ 2.°"'^^®' '^^^ 
control of funds. . 1 ' '. \ '' ' " ' 

Later, OE did improve" its Follow Through\monitor ing 
system but additional improvements in the feedback of moni- 
toring information still can be made. 

• ' /" • 

OE's monitoring system - - 

OE's program for monitoring Follow Through projects was^ 
not well defined. However, OB officials said nearly all moni- 
toring of ^oninstructional services and activities was done 
by OE project officers and with the occasional ass,rstance .of 
a consulting firm contracted by OE to provide techijical as- 
sistance to. projects. " • 

Before November 19?! OE had assigned a project officer 
to each project. "Each officer was responsible for many proj- 
ects because the number of projects exceeded the number of 
officers by more than 20 to 1. "The project officers' duties 
included approving prdject applications, providing technical 
assistance, monitoring project performance, anS monitoring 
the activities of program sponsors and the. consulting firm ' 
personnel (general Consultants). Although project • of ficers 
tried to visit each assigned project annually; it was not, 
always done. During their visits; the. officers met with 
project personnel, and parents to discuss project operations. 

To assist its project officers in their monitoring ef-- 
forts, OE assigned to each project a general consultant whose 
primary function was to visit projects each month 'to provide 
technical assistance to ^elp the project assess its strength 
atvd weaknesses. The . results- of their visits were reported to 
the OE project ofificer, the project's sponsor, and the local 
Follow Througti staff. Our review. Qf the consultants' repprts, 
for the projects we reviewed showed an averageC of 3.1 visits 
to the projects -in calendar year 1971 and 4.4 visits in 1972. 

OE'S monitoring changes — ^ . . 

From November 1971 to March 1972, OE changed its moni- ' 
toring system by. reorganizing project officer assignments 
and by providing for a desk audits and ,a team 'review at the , 
project site. ' The project officers yere organized Into five 
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ajea .teams, consisting of one , or two senior and junior 
-o^ff-icers and a secretary, with a senior officer as'' team 
leader.' Each team was' responsible .'for from 29 to 43 proj- 
ects, depending on .the size of' its geographical region, 

. The teams are to perforft an annual desk audit of each, 
project consisting of answering questions .in a standard re- 
view form concerning the various components of" the project's 
Poi;ow Through pTro.gram. information is to be obtained p>i-\ 
manly from the project's most recent application •for'^ funds. 

^n addition, OE makes 3- to 4-day onsite team reviej^s of 
projects.. The review group usually consists of a representa- 
■ tive from, the OE area , team, a geVral consultant, and an^HEW ' 
regional program specialist. 1/ ^he group uses a standard 
review form likg the de^k audit review form. Information is 
to be otL^amed from observations ,of project activities," in- 
terview^ with Follow Through staff and' parents, ahd project « 
reports and records. According to ah OE official, between 
one-third and one-half of all Pollow Through projects- are 
reviewed each year. He said that in school year 1971-72 • 
OE reviewed about 80 of the 178 existing projects. ' 

We believe that these monitoring changes, especially the 
use of standard review forms, should help improve project 
monitoring. The desJc .audits .give OE assurance that projects • 
have adequate plans' for each component', and the onsite re- 
views pr-ovide feedback da'ta.on how well =the plans are working. 

Progress and general consultant^ reports 
.could be madfe more useful • '■ • 

OE's monitoring system could be further strengthened. 
Periodically LEAs are required to submit. reports to OE on v 
th^ir progress and, :^ as previously mentioned^ the general con- 
sultants must repor't the results of their project .visits. We 
^examined a number of these reports submitted to OE on the 
projects reviewed and, rioted that they were neither • uniform , 
nox developed with a vi6w toward .improving program operations. 

1/A regional p^^ogram specialist provides v^r ious ' field serv- 
ices to State educational agencies and LEAs. ^ He devotes 
his- time to programs administered under title I of the 
Elementary and Secondary Education Act and to Follow Through 
and -IS -available to. provide field support to them and. to, - 
-he;p'OE headqu^t.fcers officials fulfill their management 
.responsibilitip|;f - • 



The grant terms and conditions require LEAs to submit an 
inte'rim and a final progress report to OE covering thS grant 
period. However OE .gave no writteji .instructions to LEAs on 
the format and content for these reports until October 1973.' 

The progress reports we review/ed were narrative descrip- 
tions supported by some statistics /of what the proj'^cts did 
during the grant vperiod to deliver (^required services. The 
reports, which varjied in .form and cjontent, did not cover all 
areas of <pfbject performance and g^erally. did not identify 
project goals or, as an alternative, present comparative^ 
statistics from the preceding year to give OE manag.ement some 
indication of the adequacy of projeqt accomplishments. They 
also contained no conclusions concerning any necessity f^r 
program improvements or corrective action. 

On September. 19, 1973, HEW published revised regulations 
on grants administration which apply to. all HEW grant pro- . 
grams. They require grantees to submit periodic performance 
reports, 'Which will include, to the extent appropriate^ a com- 
parison' of * actual accomplishments with goals established for 
the period. and, if applicable, th^ reasons goals were not 
met. However, the regulations, do not require grantees to re- 
port their conclusions concerning the,^ necessity, for program 
iitiprovements or corrective action. * . 

' Consultant report^ 

The general consultant reports, according to the con- 
sultants' manual, are to include information on problems, 
pro'gram development, areas in which consultation was and , 
should be given, and recommendations made to projects. 

The consultant reports we reviewed. Tike the progress 
reports, varied in form and. content. Although they contained 
the general information described in the consultants' manual, 
they usually did not identify specific weaknesses and probable 
causes. Such information should be included in the consult- 
ants • reports. 

^ * 

The repopts did not normally indicate whether project 
directors concurred in consultant recommendations, what proj- 
ect staffs planned to do to implement recommendations, t>r 
whet;her consultants followed -iflp on recommendations cited in 
earlier reports to determine whether they had been imple- 
mented. This type of inf orma|J^on would help insure that 
needed project improvements- ^jfe being made. 



' CONCLUSIONS 

The Follow Through projects reviewed generally had- " 
provided for the noninstructurar services and activities de- 
scribed m the program guidelines, such as organizing activi- 
ties for and employing parents/, providing for policy advisory 
committees, making 'home, contacts, giving mgdical and dental 
examinations, providing free lunches to needy; children, and 
soliciting and using community resources. 'However, adminis- 
tration of these services and activities needed improvement. 

oi .J to increase services and act>\fities and to obtain 

closer complianc^^^jw^ program requl-rea^ts was evident to 
^^Qge degree at all -projects. -A higher level of services amonq 

t^^^^f^L?5 r^^^^ 5° maximize the potential for alleviating 
the effects of poverty, and a more uniform level was needed 
to provide greater assurance that differences among projects 
in educational attainment were not due in part to differences 
m the level of services. 

Project administration needed to be improved. OE should 
(1) define, to the extent practicable, the level of perform- 
ance grantees should strive to achieve in each service com- 
H^^^i 'J^ guidance on the type of records needed to 

dpcument performance, and (3) require grantees to formally 
?nri^n ^^^'5 pro^ects. The regulations do not contain 'per- 
fnn^t. Standards or, require self-evaluation. • Rather, they 
increase the need for performance standards because they ma^e 
project refunding dependent on the extent to which projects 
provide required services and activities. 

Although OE strengthened its project monitoring durinq 
our review, the feedback of information to OE still could be 
improved. If projects and consultants were required to in- 

in their -reports the information disdussed in this chap- 
ter and to achieve greater uniformity in the form' and content 
of their reports, OE , could better monitor projects. Project 
officers could use the additional information to furnish 
technical assistance to projects.. If the reports were more 
uniform,; OE could periodically summarize the results to deter-- 
mine weaknesses common to many projects as well as -examples 
of exemplary performance. - ' ^ 

RECOMMENDATIONS TO THE SECRETARY OF HEW 

We recommend thaC-the Secretary direct_OE to: 

■ —Establish, to the extent practicable, performance 

standards for eaoh .service component and require LEAs 
to evaluate their progress toward meeting these stand- 



ards. 
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— Develop and disseminate comprehensive guidance to 
LEAS on a recordkeeping system, including the type of 
records needed to document performance and pr6vlde 
feedback 'on program results. 

, --Develop and disseminate specific guidance to LEAs ana 
consultants on the information that should be included 
in their reports to ,0E and on the format and content 
of these reports. ^ . , 

Develop, in the early implementation stage of future 
experiments, regulations and guidelines concerning 
project administration and monitoring. 

■ V ■ 

AGENCY. COMMENTS 

HEW concurred in our recommendations and said it had 
taken or planned to take the following actions: ' ' 

— OE has developed a Comprehenaive Health Service Plan 
survey which includes all the 'noninstructional Follow 
Through components.' The survey is a two-part effort 
in questionnaire form. Part one, which has been ap- 
proved by the Offipe of Management and Budget, is a 
planning questionnaire and^ will be disseminated before 
the beginning of the school year. It solicits plan- 
ning information conceri>ing services to be provided 
I to Follow Through funded children in the school year. 

It can also be used by the projects for purposes of ^ 

^ technical ^assistance by siuggesting resources and ac- 
tivities useful in delivering comprehensive health 
services. 

* • Part two, which is currently awaiting Office of Man- 
agement and Budget approval, is an end-of-year report. 
All Follow Through projects will be required to com- 
plete this report which summarizes the services ac- 
tually provided to' the children. The survey will be 
a definite means of assuring self-evaluation by the 
proj.ects in tejrms of the service components offered 
in the prograit\. • • 

— The Comprehensive Health Services Plan survey will, 
to some extent, provide guidance to LEAs on a reciord- 
' keeping ^system. In addition, OE staff members, during 
their various monitoring visits, will provide technical 

* assistance to LEAs. on the types of records rieeded to 
document program performance. 
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■OE. ,has developed a reporting form for use by Follow 
Thto.ugh- consultants which provides for additional 
information and uniformity >iTi the presentation .and 
content qf the repott. Also, the Office of Manage-^ 
ment and Budget recently approved a form which Follow 
Through grantees must use in indicating, performance 
progress. The format and content of the report are 
specific and shQpld assist the program in obtaining 

lonsistent basis. The form has-been 
ow Through grantees. 



information on 
mailed to all 



-HEW will de 
of future 
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monitoring. 




m the early implementation^tage 
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erniixg project administration and 



We reviewed the Comprehensive Health Services Planning 
Quest ion n^fljye and believe it will provide OE with e'xcellent 

)rmation concerning what services' will be or have 
jen^rpv/ided to Follo^Trttrough children. . -However, it does 
^it cdnstitiafce perforihance standards, because there are no 
requirements placed on \th^ grantees t6 provide the services 
enumerated. \We believe^OE^hould use the information ob- 
tained from the questionnairtK^o establish, to the extent 
practicable, jferformance standards which projects must meet 
to obtain refunding and which also could be used for self- 
evaluation, y 

. . V • 

We also revltewed the consultant and grantee performance 
reporting forms and believe that the consultant ■ report does 
not fully implement our recomm^dation . The report does not 
solicit information on whether project directors concurred in 
consultant recommendations, what project , staff s planned to do 
to implement recommendations, or whether ' consultants followed 
up on recommendations cited in earlier reports to determine 
whether they, had been implemented. We believe this type of 
information .should be included, to help OE better monitor 
projects. . • . ' 



CHAPTER 4 

• NEED FOR BETTER-ADMINIsAtION 

OF ELIGIBILITY REQUIREMENTS , 

' The gconSmic Opportunity Act, which, established Follow 
Through, states -that ' the program is tcTl^ocus primarily upon 
children previously enrolled in Head Start or a similar pro- 
gram, in general, the projects reviewed served enough chil- 
dren from low- income 'families .but ^not enough children with ' 
the required preschool experience. 

Inadequate project records prevented us 'from determin- 
ing the exact enrollment chai^acter istics of each project 
reviewed. In addition, OE <aid not have da!^a on the^ actual 
enrollment at Follow Through projects and lacked adequate con 
trol- over project compliance with eligibility reqairfiments 
because it: . '• . , 

--Furnished criteria to LEAs concerning the dumber and 
type of children to be enrolled, which differed from 
^ the criteria in the May 1969* memorandum of understand- 
ing between the Office of Economic Opportunity aind 
HEW. ^ 

--Did not^^urnish LEAs formal guidance on the type of 
' data to be kept to demonstrate compliance . with eligi- 
bility requirements or require LEAs- to report periodic 
cally actual enrollment data. 

—Exempted LEAs from the preschool eligibility require- 
ment without documenting its reasons. 

ELliSIBILITY CRITERIA DIFFERENCES . . .. 

* The. .May 1^(59 memorandum of understanding requires ^ that, 
with rare exceptions, at least 50 percent of the children 
participating in each Follow Thrbugh grade have preschool - 
experience, preferal^ly a full year, and come from families 
whose incomes meet Head Start income eligibility criteria 
at the time of enrollment in Head. Start. However, OE's 
draft program manual, dated February .1969, provided that, 
with rare exceptions, at least one-half^ of the children 



from low-income families 1/ in each Follow Through project 
•have aWll year of Head Start or comparable' pceschool/ 
experience. According to OE officials, the projects^ re- • 
cei,ved the manual criteria, but not the memoranduift criteria. 

,.J^^ «^"i"erences -between the two criteria are that 
U} the memorandum specifies a minimum percentage of chil- ' 
dren from low-income families' --thaft' projects must serve, 
whereas the manual contained no specification and (2) the 
memorandum provides for more children wi th. preschool expe- 
^i;!>h,-fJ project than did the manual, except when all 

•the children enrolled in a project- are from low-income fami- 

To illustrate the" differences, one project we reviewer) 
■ T/lll^^ "5-''^"'"" i>?^schc^;L^ye;r 1971-72? On the^blslt 
^•SC "'^"'°"ndum, at leW^p percent, or 153, of the 306 ' 
child^n stiould have been\r^m low^incOme" families and have 
nad sofH^ preschool experience, preferably a full y^ar . On 
the basis of the^ program manual criteria, the numblr if ' 
SS^pH r/?^"^''^^u^°,^^''^ preschool experience would be 
^^o^n 4^^^ °" lov/-income enrollment. For example, 
J itLf.n^^ children were frdm low-income famUies, 
JLi- r 50 percent, or 120,^ of them must have had" a full 
year of qualifying preschool experience. 

' , „ ^^^^"f! ^he criteria were inconsistent, we asked OE 
program officials, at the time of our project reviews, - 
rvih ^J^^u^'^u"^.''^ appropriate. The ofticikls said that, 
although they had not previously recognized the conflict 

tn"";;^ ' ? has. required, except in rare instances, 
that 50 percent of a project's enrollment consist of chil- 
dren from low-income families who have had a full year -of 
qualifying preschool experience. They said that OE wanted 
as many children as possible" from full-year Head Start 
Sa?!?™f in Follow Through because program funding 

nffeii 1 but permitted some projects to count summer" 
preschool. experience in meeting this requirement. ^ • ; 

in ^.P^.^'emorandum of understanding criterion is supported - 
in the final regulations developed for the program and 



- ?SS inioL ^ ^^"'ily was considered - - 

■ r^?lMT^n'f"^- '^-^^^ °" °^ (2) its income in' " 

relation to family size was, below "the Office of Economic 
Opportunity poverty guidelines at the time a child f?om the 
family entered preschool or Follow "Through . For example in 
December 1971 a family of four would havl been consfSered 
low-income for- Follow Th-rough if its annual . income wal under 
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published in the Federal Register bn April 21, 1975. ' 
These regulations state that: 

'•At least 50 percent of the children in each 
entering class shall be children who have pre- 
viously participated in a full-year Head Start 

.or similar quality preschool program and who 
were low-income children at the time of enroll- 
ment in such preschool program; e^ccBpt that the 
Commisjsioner ,may reduce this percentage require- 
ment in spe<:ial cases where he deteritiines that 
its enfarcement would prevent the most ef fec- 

' tive use of Follow Through funds. * * 

Did project enrollments meet the criteria ? ' r' 

fTTo deteriftfne the 'eli^ib^i^^char^cter istiee-*^ projeclr%r^ 
e/irolle^s, we selected at random two. groups of children at 
each project for comparison'. The -first group 'represented 
children enrolled during school year 1970-71, and the second 
group represented the 1971-72 enrollment.,^ We select^fl at 
leaat 100 children for each group, bat, at' 2 projects we ^ 
reviewed the entire enrollment for both school years. 

For, each child sample^, we attempted to determine' from 
available records (1) the type and length of the child •s 
preschool exper^ience, (2) annual family income and the pe- 
riod during -which this income was earned^ (3) family size, . 
and (4) family wpl|||re status^ -Our examination included 
Aonproj^ect records, such as attendance rolls of loCal Head 
Start centTers and "applications for free lunches under the 
Department of Agriculture* s 'National, School Lunch Program. 

Our sample rel^ults. 'f rom eight of the nine projects are 
shown in the following three charts. The odd-numbered sam- 
ples are for^ school. yeay 1970-71^ and the even for 1971-72. 
One project was not included* because we could not reconcile 
our statistics with those claimed by the project. 

The .charts ^how that 7' of ^ the 8 projects served, enough 
children from low-income families (chart 1) but that the 
.preschool characteristics of their enrollments were 

— beiow that intended by, the memorandum of uhderstand- 
ihg- as interpreted by OE in all but 1 of the 15 sam- 
ples for wKich records were available (chart 2) and 
, - /• > 

— b*elow the manual -cr iter ia in 5 of the 15 samples for 

, -which records were availabl-e (chart 3). 




CHART 1 

CHILDREN FROM LOV-^iNCOHE FAMILIES 



, PEKCENT 




PIRCEMT OF TOTAL CHt^DftEN kAMPLCD FROM LOW-INCOME FAMILIES. 
fWy^ THE OpD.^MMKED SAMPLES iEPRESENT SCHOOL YEAK H70--71 AND THE.EVEN.NUMIEREO 
ISTaMPLES represent school yHr 1971-72. 
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CHART 2 

MEMORANDUM OF UNDERSTANDING CRITERIA 
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CHITEHU LINK— INOICAT« A f HOJECrS 

eNROLLMKKT M8T THE Af f LiCAiLE tRITERIA. 



PERCENT OF TOTAL CHILDREN SAMPLED FROM LOW-INCOME FAMILIES "^^J^^l^^'^f*;^ 
rmWl PRESCHOOL experience. THE.0OO.NUMIEREO SAMPLES REPRESENT SHOOL YEAR inO.^rl 
f^M AND THE EVEN-NUMBERED SAMPLES REPRESENT SHOOL YEAR lt71-72. 
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CHART 3 . 
PROGR^ MANUAL CRITERIA 
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CRITERIA LINE— INDICATES WHETHER A PROJECrS 
ENROLLMENT MET THE APPLICAiLE CRITERIA. 



PERdENT^OF. CHILDREN FROM^OW-lNCOME FAMILIES IN BACH SAMPLE WITH FAILL^YEAR 
THE ODD^NUMBERED SAMPLES REPRESENT SCHOOL YEAR 1970. 



-PRESCHOOL EXPERIENCE _ 

F i tffifl'nl AND THE EVEN^HUMBERED SAMPLES REPRESENT SCHOOL YEAR-1971.7^ 
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•■ ■ - ■ ■ ^, . ■• ; .. " 

Because of inadequate records, wa cou-ld not determine, 
the income status of about 13. percent of the children sam- 
pled and the preschool status..of. aboiJt 8 percent. •- „ 

Regarding the' low proportion of children from low-income 
families who had a full year of preschool experience, OE _ 
program officials told us that the following factors influ- 
enced the number o£ full-year preschool, children who fcould 
attend Follow Through: 

1. Many Head Start projects in the Sl^th, Where io of 
our 16 samples were taken',, were summer projects : and 
were in areas not having the highest concentrations 
of low-income families, whereas Follow Through- tried 
' to establish its projects in the most poverty-stricken 
areas.' • ' 

'2. Head Start consolidated projects in many parts of'the 
Nation, causing a reduction in Head Start enrollments 
in some Follow Through' project areas. 

• -3.' Many Southern school systems were implementing deseg- 
. legation plans,' causing^spersion of eligible . chil- 
dren. 

4. Follow Through lost some students through attr itrren^- 

' The officials explained thaf support was not withdrawn 
from pr6j"ects^affeited by these factors' bedause such action 
would have- disrupted the Follow Through experiment. 

An^OUAC Y OF J>ROJECT 'eligibility RECORDS ■ :^ 
AND FEEDBACK OF ELIGIBILITY DATA TO OE ^ ^ 

OE did' not require projects to keep enrollment records 
or furnidh ^hem formal guidance on recordkeeping procedures. 
OE also did not require periodic reports of actual enrollment - 
'data, in addition,' some children's family income ^tatus had 
.been improperly determined. For example, the project with 
the highest incidence of improper determinations had classi- 
■fied 153 children .in school yearh970-71 and 223 children 
the following school ye^r as coming from low.-incpme families. 
From a. reviev of the enrollment data for all chlldr^n m the 
oroiect, we dete^^ined, and the project director concurred, 
that 37 and -70 children, 'respectively, had. been erroneously 
classified as- from low-income families and, 29 and 12 children, 
jifst as erroneously, had been classif ied .as non-low-income. 
The number 'of improper classifications varied among the pro:- 
ects ?rom none tfmany. One project made- no f^^empt to clas- 
sify children by family income status and anotl^r did not 
'..classify Its children yptil the fourth year of program. 



Most Of the observed *proceduraI'^ weaknesses were due'* to 
^eifeher the^ enrollment or other eligibility forms nOV being 
completely filled out X)k their design not providing all the. 
infbrmation needed to d^terrafihe whether a child was, from 
the target population. The^poorXy designed forms generally 
lacked information .on length and type of preschool experience 
arid family welfare sta.tus. A family on welfare automatically 
qualified' for the f^ll range of Follow Through services'.' 

0? program officials told us in September 1973 that they 
cannot prescri^be uniform procedures for collecting enrollment 
data because thfe -collection of personal information is highly 
sensitive and generates strong concerns about invasion of pri- 
vacy.' They said, however, that they plan to dissem'inate guid- 
ance to projects on the format for recording enrollment data. 

OE officials Hso said they plan to ..develop a procedure 
to collect actual enrollment data from the .projects: As of 
April 1974, almost all OE's information on the number and ^ 
type of children enrolled in Follow Through wag based on . - 
estimates included' in the proj-eet applications for funds. 
In October 1972 OE did make a one-time request of projects 
for actual enrollment data. Our; analysis of the information 

-submitted to' OE showed that 82 percent of the children en- 
rolled in Follow, yhrough were from low-income families. OE 
did npt request information on how many children attended 

jfull-year preschool programs. - 

In our opinion r OE's plans to provJLde guidance to proj- 
ects and to collect actual enrollment da^a are needed to in- 
sure that the prograiri's target population is served. \ . ' 

WAIVER OF PREfiCBOOL REQDIRE-MENT ' ^ 

* According to OE program of ficiais , waivers w6re made in 
bhe^form of their recommendation to. an HEW contracting officer 
that a project be funded. Thus^ in those instances where a 
w-aiver was granted, the program' requir-ements were waived in 
favor o^ the provisions in the application. Cfe .exempted sev-- 
eral LEAs from meetixig the Follow Through preschool reqaire- 
ment but did not document the reasons. 
'> * 

"^--l Of the 18 approved applications we reviewed for school 
years 1970-71 and 1971-72, 5 shbwed that, the'planned enroll-- 
ment of' children from low-income families with* any preschool 
experience at ail ;was below 50* percent of the total enroll- 
ment, as required. At our request, an. OE program official 
identified = 19 projects in school year 19'^2^73 with prcf;}i« 

.Head ^tart enrollments below 50 percent, of the projected 
total enrollments. / , . , - 



We could not deter mime, -whether the r^^sohs for the 
waivers of the preschool reqdireinent were, valid because they 
were not documented. OE program officials agreed th^t such . \ 
documentation should have been required; that in the early 
years Follow- Through' s staff devote.d its efforts to resolv- 
ing complex issues involved in starting an innovative pra- 
gr^m^^and that recordkeeping was not given high priority* 
The officials felt that althoijgh a written record was not 
made/ each application was reviewed in depth and waivers ; 
of program requirements were thoroughly discussed in face- ^ 
to-fac?e negotiations 'with project staff. * * • . 

CONC LUSIONS , ^ , . ' 

OE's administratJon of 'project compliance with eligi- 
bility requirements could be improved to insure. that the 
intended target population is served. The actual mimber 
of eligible 'children in Follow. Through could not be accurately 
determined because/of the poor quality of praject eligibility 
records and the lack o'f a procedure requiring periodic eligi- 
bility reports from the' projects .* * 

On the basis of our. sample results and- OE data/ the 
projects enrolled enough children from low-income families 
to meet the't>rogram criteria/, but many^ fell^short of the 
.goal of serving childrea from full-year Head Start projects, 
partially- because OE waived this requirement* OE's reasons 
may have been ^vali^V't5^U: document^ation was ncrt available to 
show .the alternatives/ ih aoy/ to ijranting waivers. - ^ 

RECOMMENDATIONS TO THE SECI^TARY OF HEW > 

We recommend that, the Secretary direct- OE to: 

— Provide , guidance to LEAs on the format of eligibHi^ty 

records and require them to periodically report actual 

eni?olljnent data," as planned. ; - " 

... • ' , 

—Document reasons for exempting LEAs from program eli- 
gibility <requirement3 . ^ /. ' : 

AGENCY COMMENTS 

HEW concurred in our recommendation^ an'd said: 

— *0E will begin developing a format which XEAs could ; * * 
use to document 'th? eligibility of children and will 
attempt to .obtain Office of .Management and budget 
' approval *for .a periodic report .of actual exiroj-lment . 



-OE will take Steps to assure documentation in/ those 
instances, where, for justifiable reasons, Joillow 
•Through program regulation requirements are waived. 
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DEPARTMENT OF HEALTH. EDUCATION. AND WELFARE 
OFFICE OF THE SECRETARY 
WASHINSTON. O.C. 20201 



'•^L 2 6 1975 



Mr. Gregory 0. Ahart 
^r^cto^sJ^anpower and 
Wei fareuli vision 
U.S. General Accounting Office 
Washington, D.C. 20548 

Dear Mr. Ahart: 

The Secretary asked that I respond to your request for our consents 
on your draft report to the Congress entitled, "Follow Through: 
Lessons Learned From its Evaluation and Opportunities to Improve its 
Administration". They are enclosed. 

We appreciate the opportunity to comment on this draft report before 
its publication. 

—Sincerely yours, 
D. vOung* ( 

stant Secretary, Comptroller 




Enclosure 
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Xomments of the Oepartment of Health, Education, and Welfare on the 
jCocnptron^r Genera rs Report to the Congress Entitled, "Follow Through: • 
llessojis Lisarrned From its Evaluation and Opporturiltles^to Improve fts 
g^dmini strati on" 



GAP Recommendation ' 

Develbp greater safeguards to irfsure that future experimental programs 
are got designed apart from evaluation to maximize the degree to 
^ which experimental results will 6e statistically reliable. 

Department Comment ^ 

We concur with the recommendation and will try to insure that 
legislation and r^egulations for future experimental programs 
are consonant with strong evaluation designs by requiring 
random assignment of projects to school districts, schools, 
classrooms* etc., and requiring school districts to establish 
and maintain comparable control groups or schools. The Pffice 
of Education will formally advise all program offices of the 
importance of the making evaluation an integral part of 
experimental programs. 

GAP Recommendation 

Translate the problems encountered during the Follow Through 
experiment into criteria for designing and administering ^ 
future experiments, includ1nq> among other things, criteria 
to prevent massive data collection on program results before 
the program's goals and objectives are properly deflated . 

Department Comment 

We^^ree. The .(experience of Follow Through- re-emphasizes the 
need for planning prior to implementation of new program designs 
or major change in basic program thrust. Although OE does not 
frequently undertake experiments like Follow Through,' we recognize 
that some of the Follow Through experience 1s pertinent to other 
program evaluations* We now have a-centralized PE evaluation staff 
Responsible for developing evaluation designs based upon program 
^objectives so as to prevent unnecessary data collection. Jhis 
staff is cogn'izant of the problems encountered and will to* the 
•extent possible incprporate this recommendation in the overall 
•: evaluation designs. * T ' 
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GAP Recommendation 

£s.tab-1Uh, to the extent practicable, performance standards 
for each service componerit and require LEAs to evaluate 
their progress toward meeting these standards. 

Department Comment / 

.We agree with this recommendation, and to some extent, this 
recommendation has been Implemented. OE.has developed a handbook 
entitled, "Handbook on Perforniance Objectives, Title I, Elementary and 
Secondary Education Act of 1965" and Follow Through 1s included as onfe 
of the programs for which this handbook would be useful. .Also, OE 
has developed a Comprehensive Health Service Plan, survey which , 
Includes all Follow Through components except the Instructional related 
component. This survey Is a two part effort and 1s in the form of a 
questionnaire. Part one, "Follow Through Comprehensive Ileal th Services 
Planning Questionnaire" has been approved by 0MB and will be disseminated 
prior to the beginning of the school year, ft solicits planning Infor- 
mation concerning services to be provided to Follow Through funded children 
In the school year. In addition to summarizing plans for delivery o-^ services 
the questionnaire can be used by the projects for purposes of technical , 
assistance by suggesting resources or activities which could be useful in 
the delivery of Comprehensive Health Services. Part two. of the survey is an 
end-of-year report, "Follow Through Comprehensive Health Services End-of-Y6ar 
Report". It is currently awaiting 0MB approval, and Is a means of noting x 
reject accomplishments for a recently completed school year.. All Follow 
hrough projects will be required to complete this report which summarizes . 
the actual services provided for the Follow Through children. This survey 
win be a deffftjte means of assuring self-evaluation by the projects 1n 
terms of the service components offered in the program. 

GAO Recommendation 

Develop and disseminate to LEAs comprehensive guidance on a 
recordkeeping system. Including the type of records needed to 
document performance and provide feedback on program results. 

Department Comment * 

Ve agree with this recommendation. OE has to some extent accomplished 
the implementation of this recommendation with the "Comprehensive 
Health Services Planning Questionnaire", OE Form 4535, and the <iraft 
"FoUow Through Comprehensive Services End-of-Year Report". To the 
extent possible, OE Staff members during their various monitoring 
Visits will provide technical assistance to LEAs 1n the types of 
records needed to document program performance. 
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GAP Recoffwendatlon ' 

Develop and disseminate specific guidance to LEAs and consultants 
on the infortnatioji that should be "included in their reports to OE 
and on the format and content of these reports. 

Department Comnent. . , . . 

• 

This recomnendation has been implemented. OE has developed a 
reporting form for use by the Follow Through consultants which • * 
provides for additional informatton and untfSrmity in the 
presentation and content of tfie reports as suggested by 6A0. • 
Also 0MB has recently approved a form which grantees must use in 
Indicating progress in performance of the Follow Through grants. 
The format" and co^itent of the report are specific and should 
assist the program Xn obtaining information on a consistent basis. 
This form has been meitled to-a+l-FoMow Through- grantees . 

GAP Recommendation 

Develop, in the early implementation stage of future experiments, : 
regulatTdns and guldelTnes concernT'ri^ prefect administ ration and 
wonnoring. ■ — ' 

Departnent Comment 

We agree with the reconmehdation and it will be implemented in 
future experimental programs. 

CIAP Recommendation 

Provide guidance to LEAs on the' format of eligibility records 
and reguire them to^j)eriodicaily report actual enrolment date. 

Department Comment . ' • 

■■■■ We agree with the recommendation. OE will begin developing a 
;, format which local project grantees could use to document the . > 

eligibility of children and will, again attempt to obtain approval 
from^OMB for the report of actual enrollment periodically. 

GAt) Recomnendation .-. * ' . • 

^ Document reasons for exempting LEAs from program eligib ility 
reguirements. =~" 

DepartmentwConment 

We agree with this recommendation. OE will take steps, to assure 
documentation in those instances where for Justifiable reasons 
follow Through program regulations requirements are being waived. 

- \ ' ' 62 ' - . ^ . ■ • 
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GAO no%tjl The Mterial on the remaining two pag«s w«s deleted 
'.^ * ' because it relates to general matters i^ich were 
considered or incorporated into the final report. 
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PRINCIPAXi HEW OFFICIALS 
RESPONSIBLE FOR THE ACTIVITIES 



DISCUSSED IN THIS REPORT 



SECRETARY OF HEALTH, EDUCATION, 
AND WELFARE: 

David Mathews . 
Caspar W. Weinberger . ' 
. Frank, C. Carlucci (acting) 

Elliot L. Richardson 
; Robert H. Finch 
Wilbur J. Cohen 
John W. Gardner 

ASSISTANT SECRETARY (EDUCATION): 
Virginia Y. Trotter 
''Charles B. Saunders, Jr. (acting) 
Sidney P. Mar land, Jr. 

COMMISSIONER OF EDUCATION: 
Terrel H. Bell^ 
John R. Ottioa 
John R. ottina (acting) 
Sidney P. Marland, Jr. 
Terrel H. Bell (acting) 
James E. Allen, Jr. 
Peter P. Muirhead (acting) 
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Copies of C^AO reports or« ovoMobie to the generol public ot 
a Cost of $U0O o copy* Th«r« is no chorg« for. reports furnished 
to Members oT Congress ond congressionot committee ,stbif 
members; officiols of Federol, Stote, locol, ond foreign govern* 
ments; members of the press; college llbrories, foculty members, 
ond students; ond non*profit orgonizotions. 

Requesters entitled to reports without tharge should oddress 
their requests to: 

^ ' • U«S* Generol Accounting Office 

^ Distribution Section, Room 4522 
. 441 G Street, NW. 
' Washington, D.C. 20548 ' 

Requesters who ore required to pay for reparts should send 
their requests with checks or money orders to: 

U.S. General Accounting Office « 
Distribution Section , 
P.O; Box 1020 s ) ^ 

Washington, D,C. 20013 , 

Checks or money ^orders should be mode poyoble to the 
U«S« Generol' Accounting Office* Stomps or Superintendent 
of Dbcuments coupons will not be occepted. Pfeose do not 
send cosh. 



To expedite filling your qrder/ use the report number in the 
tower left corner of the front cover. 
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